Method for classifying test subjects in knowledge and functionality states

ABSTRACT

A method for classifying a test subject in one of a plurality of states in a domain, a domain being a set of facts, a quality measure, or a combination of the two. The set of facts for a knowledge domain is any set of facts while the set of facts for a functionality domain is a set of facts relating to the functionality of a test subject. A state is characterized by a subset of facts, a value for a quality measure, or a combination of a subset of facts and a value for a quality measure. A first state is higher than or equal to a second state and a second state is lower than or equal to a first state if (1) the subset of facts or the quality measure value associated with the first state respectively includes the subset of facts or is greater than or equal to the quality measure value associated with the second state or (2) the subset of facts and the quality measure value associated with the first state respectively includes the subset of facts and is greater than or equal to the quality measure value associated with the second state. Decision-theoretic rules are specified for selecting the test items to be administered to a test subject, for determining when it is appropriate to stop administering test items, and for determining the classification of the test subject. A test subject is classified in the highest state of which he has the knowledge or functionality.

BACKGROUND OF THE INVENTION

This invention relates generally to methods and systems for testinghumans and systems and the subsequent classification of humans intoknowledge states and systems into functionality states. Morespecifically, the invention relates to computer-implemented testing andclassification systems.

The process of testing and classification requires meaningful andaccurate representations of the subject domain in terms of domainstates. The domain state that a test subject is in is determined bysequentially administering to the test subject test items involvingdifferent aspects of the subject domain. The responses of the testsubject to the test items determines the state of the test subject inthe subject domain.

The implementation of such a testing and classification process by meansof a computer has the potential of providing an efficient and effectivemeans for identifying the remedial actions required to bring the testsubject to a higher level of knowledge or functionality.

The partially ordered set ("poset") is a natural model for the cognitiveand functionality domains. Two states i and j in a poset model S may berelated to each other in the following manner. If a test subject instate i can respond positively to all the test items to which a testsubject in state j can but a test subject in state j may not be able torespond positively to all the test items to which a test subject instate i can, we say that i contains j and denote this by the expressioni≧j. Note that a positive response on any item should provide at leastas much evidence for the test subject being in state i as in state j.Thus, the domain states are partially ordered by the binary "i containsj " relation. Note that the cognitive level or the functionality levelof a test subject in state i is equal to or higher than that of a testsubject in state j. Similarly, the cognitive level or the functionalitylevel of a test subject in state j is equal to or lower than that of atest subject in state i. Accordingly, state i is said to be equal to orhigher than state j and state j is said to be equal to or lower thanstate i.

Poset models in an educational context have been proposed before.However, they have either been Boolean lattices or posets closed underunion in the sense that the union of any two members of the poset isalso in the poset. This restriction is undesirable in that it leads tomodels that can be quite large. For example, allowing the number of testitems to define the model can lead to models with as many as 2^(N)possible states where N is equal to the number of test items. With thisapproach the responses to the test items permits immediateclassification with very little analysis. However, having such overlylarge models ultimately results in poor classification performance.

When sequential item selection rules have been used in classifyingstates in a poset, the approach has not been accomplished in adecision-theoretic context. Consequently, there was no assurance thatthe classification process would converge rapidly nor, in fact, that itwould converge at all.

There is a need for a testing and classification system which is basedon sound scientific and mathematical principles and which, as a result,can accurately and efficiently determine the domain states of humans andsystems. It is reasonable to base such a system on poset models, but itshould be possible to use general, even non-finite posets rather thanthe specialized posets that are typical of present-day systems. It isimportant that model selection and fitting for any particular domain bebased on appropriate analysis rather than simply a result of the choiceof test items. Similarly, the selection of test items should be based onappropriate analysis with reference to the domain model rather thanbeing a more-or-less ad hoc process that ultimately gives birth to itsown domain model.

SUMMARY OF THE INVENTION

The invention is a method for classifying a test subject in one of aplurality of states in a domain, a domain being a set of facts, aquality measure having a range of values, or a combination of a set offacts and a quality measure. The set of facts for a knowledge domain isany set of facts while the set of facts for a functionality domain is aset of facts relating to the functionality of a test subject. A state ischaracterized by a subset of facts, a value in the range of values for aquality measure, or a combination of a subset of facts and a value for aquality measure. A first state is higher than or equal to a second stateand a second state is lower than or equal to a first state if (1) thesubset of facts or the quality measure value associated with the firststate respectively includes the subset of facts or is greater than orequal to the quality measure value associated with the second state or(2) the subset of facts and the quality measure value associated withthe first state respectively includes the subset of facts and is greaterthan or equal to the quality measure value associated with the secondstate. A test subject is classified in the highest state of which he hasthe knowledge or functionality.

The first step of the method consists of specifying a domain comprisinga plurality of states and determining the higher-lower-neitherrelationships for each state, the higher-lower-neither relationshipsbeing a specification of which states are higher, which states arelower, and which states are neither higher nor lower. The plurality ofstates includes a first, second, and third fact state characterized bysubsets of facts wherein (1) the first and second fact states are higherthan the third fact state and the first fact state is neither higher norlower than the second fact state or (2) the first fact state is higherthan the second and third fact states and the second fact state isneither higher nor lower than the third fact state.

The second step of the method consists of specifying a test item poolcomprising a plurality of test items pertaining to a particular domain.

The third step of the method consists of specifying an initial stateprobability set (SPS) for the test subject to be classified, each memberof the initial SPS being an initial estimate of the probability densityvalue that the test subject is associated with a particular state in thedomain.

The fourth step of the method consists of specifying a class conditionaldensity for each test item in the test item pool for each state in thedomain, a class conditional density being a specification of theprobability of a test subject in a given state providing a particularresponse to a specified test item. Each test item partitions the domainof states into a plurality of partitions according to the classconditional densities associated with the item, a partition being asubset of states for which the class conditional densities are the sameor the union of such subsets.

The final steps of the method are administering one of a sequence oftest items from the test item pool to a test subject and, afterreceiving a response to an administered test item, updating the SPS anddetermining whether to stop the administering of test items and classifythe test subject.

Decision-theoretic rules are specified for selecting the test items tobe administered to a test subject, for determining when it isappropriate to stop administering test items, and for determining theclassification of the test subject.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example of a simple poset model as a Hasse diagram.

FIG. 2 shows a flow diagram for the process executed by a computer inclassifying a test subject and providing remediation in the case of ahuman test subject or remediation guidance in the case of a system testsubject.

FIG. 3 shows the flow diagram associated with one embodiment of theclassification step shown in FIG. 2.

FIG. 4 shows a portion of a strategy tree embodiment of theclassification step shown in FIG. 2.

FIG. 5 shows the relationship between the loss function and a strategytree.

FIG. 6 depicts a poset model as a Hasse diagram.

FIG. 7 shows the image of the mapping of a test item pool on the posetmodel of FIG. 6.

FIG. 8 depicts a more complicated poset model as a Hasse diagram.

FIG. 9 depicts the poset model of FIG. 8 with a missing state.

FIG. 10 depicts the poset model of FIG. 8 with many states missing.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The first objective of the present invention is to provide meaningfuland accurate representations of a cognitive domain in the case of humansand a functionality domain in the case of systems (where "systems"includes humans considered as systems) through the use of partiallyordered sets (posets). The second objective of the present invention isto provide a method for efficiently and accurately testing andclassifying humans into cognitive domain states and systems intofunctionality domain states. The third objective is to provide aremediation program keyed to a domain state and designed to bring ahuman or a system to a higher domain state.

The classification process consists of administering a sequence ofresponse-generating test items to the test subject. The responses to thetest items provides the means for classifying the test subjects intoparticular domain states. In the case of humans, the test items might bequestions or problems relating to particular subject matter such asarithmetic, chemistry, or language. In the case of a system, a test itemmight consist of (1) causing a system to be in a particular operatingstate and (2) causing certain inputs to be entered into the system, theresponse to be observed being the operating state of the system afterthe specified inputs have been entered into the system. The operatingstate and the functionality state of a system are two differentconcepts. For example, if the system is an aircraft, the operating statecould be defined as its position and attitude and the time rate ofchange of its position and attitude. Its functionality state is ameasure of the degree to which each of the functions that must beperformed within the aircraft system to maintain it in an operatingstate are being performed.

The classification capability of the system provides the means forprescribing remedial programs aimed at propelling humans and systemsinto higher and higher states in their respective cognitive andfunctionality domains.

The poset models which provide the basis for the present invention maybe either finite or non-finite. In many educational applications, it canbe assumed that the poset is finite with top and bottom states denotedby 1 and 0 respectively, 1 denoting complete knowledge or functionalityin a particular domain, and 0 denoting essentially no knowledge orfunctionality. However, this invention applies just as generally toposet models without a top and/or bottom state. For much of the materialto follow, it will be assumed that the underlying model is a discreteposet. Later, a non-discrete poset model will be described.

A formal definition of a partially ordered set and a partial order is asfollows. Let P be a set with a binary relation ≦. P is said to be apartially ordered set, and the binary relation ≦ a partial order if forall elements i, j, and k in P the following conditions hold: i≦i, i≦jand j≦i implies i=j, and i≦j, j≦k implies i≦k. an example of a poset isin FIG. 1, depicted as a Hasse diagram. Note A≦1, B≦1, 0≦A, 0≦1, etc. Ifi≦/j and j≦/i, i and j are said to be incomparable, and the relation"strict inequality" is said to occur on P when i≦j and i=/j.

Associated with each test item and each domain state of a test subjectis a class conditional density f_(i) (x|s) which is the probability of astate-s test subject providing the response x to a test item i. Forsimplicity, test items are assumed to be conditionally independent inthe sense that responses to previously administered items do not affectthe response to a presently-administered item. It should be noted,however, that this assumption can be relaxed, and all the techniques tobe described below can be applied to the case where the test items arenot conditionally independent. Moreover, response x can be viewed asmulti-dimensional without loss of generality.

For a properly structured poset model S and properly designed testitems, a test item will partition S into subsets according to the classconditional densities associated with the test item. It will be assumedfor now that a test item partition consists of the subsets of stateswhich share the same class conditional density. In practice, an expertmust initially specify the partitions according to how he believes theresponse distributions should be structured and possibly shared amongstates. Specification and/or estimation of the class conditionaldensities can then be conducted. (The estimation process will bedescribed below.) Modification of the partitions is possible afterfeedback from data analysis, as will also be described below.

One of the subsets may be a principal dual order ideal (PDOI) generatedby a state in S. The PDOI generated by a state s is the set of states {jin S:s≦j}, where ≦ denotes the partial order relation. A commonpartition for an item will have two elements, one being a PDOI,generated say by s in S. Under such circumstances, the test item is saidto be of type s or associated with s. A test subject in the PDOI ofstate s is more likely to provide a response to a test item of type sthat is consistent with the knowledge or functionality of state s than atest subject who is not.

More generally, reference to an item's type refers to its partition.Note that one of the partitions could be the union of PDOIs.

The system works best when the response distributions reflect theunderlying order structure on the model S. This can be achieved forinstance by imposing order constraints on item parameters associatedwith corresponding item partitions. For example, with Bernoulli responsedistributions, it maybe natural to assume f_(i) (X=1|s₁)≦f_(i) (X=1|s₂)for item i if s₁ ≦s₂ in S where X=1 implies a positive outcome where weuse the term "positive" in the sense that the outcome is more consistentwith the knowledge or functionality of state s₂ than that of state s₁.The system works most efficiently in applications when such orderconstraints on the class conditional response distributions are naturalfor the underlying poset model.

Clearly, each state may have its own response distribution for an item.However, in practice, this may present a difficult estimation problemsuch as having too many parameters. Hence, using the minimum number ofnatural partitions for an item is desirable for simplifying the densityestimation process.

In the educational application with Bernoulli responses, a naturalpossible two-element partition for an item is the subset of states in Swherein test subjects have the knowledge or functionality to provide apositive response to the item and the complement of the subset. It isnatural to assume that the probability of a positive response by testsubjects in this subset of states to be greater than that for testsubjects in the complement. Further, specifying one of the subsets as aunion of PDOIs can reflect that there exists multiple strategies toobtain a positive response.

The partially ordered set structure of domain states permits greatefficiency in classification of test subjects. For example, suppose theset S consists of the states 0, A, B, C, AB, AC, BC, ABC (=1) where thecognitive or functionality domain is divided into three areas A, B, andC. The symbol 0 denotes no knowledge or functionality. The symbols AB,AC, and BC denote knowledge or functionality equivalent to the unions ofindividual area pairs. And the symbol ABC denotes knowledge orfunctionality equivalent to the union of all of the individual areas.Assume the item distributions to be Bernoulli, with the probability ofpositive responses given that the test subject has the knowledge orfunctionality to provide a positive response to be 1, and theprobability of a positive response given that he does not to be 0.Administering a test item of type A partitions the set S into the PDOIof A (i.e. A, AB, AC, ABC) and the complement (i.e. 0, B, C, BC). If thetest subject gives a positive response, a test item of type B partitionsthe PDOI of A into the PDOI of B (i.e. B, AB, BC, ABC) and thecomplement (i.e. 0, A, C, AC). If the test subject again gives apositive response, we have narrowed down the possible states for thetest subject as being the intersection of the PDOI of A and the PDOI ofB or the set consisting of AB and ABC. If the test subject now gives anegative response (i.e. not a positive response) to a test item of typeABC, we have determined that the test subject should be classified instate AB. Thus, by administering only three test items, we have managedto uniquely classify a test subject into one of 8 possible states. Ingeneral, the classification process becomes more complex as the responsedistributions become more complex in the sense that there may exist avariety of possible responses, not all of which are statisticallyconsistent with the true state identity.

The basis for the computer-implemented testing and classificationprocess is a poset model S and a test item pool I. The statisticalframework used to classify test subject responses is decision-theoretic.This entails selection of a loss function to gauge classificationperformance. In general, a loss function should incorporate a cost ofmisclassification and a cost of observation. For a given test subject,an initial state probability set (SPS at stage 0) is assigned as well,and denoted as π₀. The SPS at stage 0 consists of prior probabilitiesconcerning the test subject's state membership in S, and there exists inthe set for each state s in S a prior probability value π₀ (s). Thedecision-theoretic objective in classification is to minimize anintegrated risk function.

There are three main issues in classification: item selection, decidingwhen to stop the item administration process, and making a decision onclassification once stopping is invoked. We define a strategy δ to bethe incorporation of an item selection rule, stopping rule, and decisionrule. What is desired is to find strategies that minimize the integratedrisk function R(π₀,δ) which will be defined later. For a description ofthe framework when S is finite, see J. Berger, Statistical DecisionTheory and Bayesian Analysis, Second Edition, Springer-Verlag, New York,1985, p. 357.

As mentioned earlier, loss functions should incorporate a cost ofmisclassification and a cost of observation. Whether a decision rulemisclassifies depends on which state is true. Hence, the system assumesthat loss functions depend on the true state s in S, a decision ruled(x_(n)) which is a function of the response path x_(n) of length n, andn, the number of observations. Note that d(x_(n)) can be viewed as afunction of the final SPS through x_(n) and the initial SPS, π₀. A lossfunction may be denoted by L(s,d,n) where d is the action that thedecision rule d(x_(n)) takes. Being a function of n includes the casewhere test items have their own cost of observation.

In order for a loss function to encourage reasonable classification, itwill further be assumed that for fixed s in S and fixed number ofobservations n, when the decision rule takes an action that results in amisclassification, the value of a loss function will be greater than orequal to the value if the classification decision was correct.Similarly, for fixed s in S and fixed classification decision, the valueof a loss function will be non-decreasing in n, the number ofobservations.

Besides serving as objective functions to measure the performance of theclassification process, such loss functions are used in stopping rules,generating decision rules, and item selection.

Given a loss function and initial SPS, it is desired to find a strategy67 which minimizes ##EQU1## where f(x_(N) |s) is the responsedistribution for possible response path x_(N) of length N for a givenstate s in S, N is random and dependent on the response path, itemsequence is selected by δ and the stopping rule is given by δ, and theclassification decision rule d(x_(N)) is given by δ. This quantity isknown as the integrated risk of δ given the initial SPS. It is thecriterion on which to base the performance of strategies in theclassification process. If the possible responses are continuous, thenone would integrate as opposed to sum over all possible response paths.When N=0, equation (1) gives the average loss with respect to theinitial SPS.

A linear version of a loss function is given by the equation ##EQU2##where L(s,d,N) is the loss function associated with the action ofclassifying in state d a test subject whose true state is s afteradministering test items i₁, i₂, . . . , i_(N). The constants A₁ (s) andA₂ (s) are the losses associated with correct and incorrectclassifications respectively for state s being true. Assume A₁ (s)≦A₂(s). This relation signifies that the loss associated with a correctassignment is always less than or equal to the loss associated with anincorrect one. The cost of administering a test item C(i_(n)) suggeststhat the cost may be a function of the test item. For example, thecomplexity of items may vary and the cost of developing andadministering the items may vary as a result. For simplicity, the costof administering a test item can be assumed to be the same for all testitems.

For purposes of discussion, let us assume that C(i_(n))=0.1 (a constant)for all test items and that A₁ (s)=0, A₂ (s)=1 for all states s in S.Suppose at stage n, the largest posterior value in the SPS is 0.91. Theoptimal decision rule for this loss function in terms of minimizing theintegrated risk given a response path and given that stopping has beeninvoked is to take the action of classifying to the state with thelargest probability value in the final SPS. An optimal decision rule interms of minimizing the integrated risk is referred to as the Bayesdecision rule. With respect to this loss function and the correspondingintegrated risk, it is not worth continuing since the reduction inaverage misclassification cost cannot possibly exceed the cost of takinganother observation. If C(i_(n)) were equal to 0 for all test items, itwould presumably be worth continuing the administering of test itemsindefinitely in order to obtain higher and higher probabilities of acorrect assignment since the cost of an incorrect assignment overpowersthe nonexistent cost of administering test items. This example gives anindication of how the cost of observation plays a role in deciding whento stop, and how the cost of misclassification and cost of observationmust be balanced.

The basis for the computer-implemented testing and classificationprocess is a poset model S and a test item pool I that is stored in thecomputer memory.

Consider again the poset model in FIG. 1. For the cognitive domain ofarithmetic, state A might represent a mastery of addition andsubtraction, state B might represent a mastery of multiplication anddivision, and state 1 might represent a mastery of arithmetic, the unionof states A and B. For the functionality domain of an AND gate, state Amight represent the proper functioning of a NAND gate, state B mightrepresent the proper functioning of an inverter which inverts the outputof the NAND gate, and state 1 might represent the proper functioning ofboth the NAND gate and the inverter (i.e. the AND gate).

The flow diagram 1 for the process executed by the computer inclassifying a test subject and providing remediation in the case of ahuman test subject or remediation guidance in the case of a system testsubject is shown in FIG. 2. The process begins with the initializationstep 3 whereby the poset model, the test item pool, and the testsubject's initial state probability set are stored in computer memory.

The poset model is defined by a list of states, the PDOI for each state,information about the costs of correct and incorrect classification foreach state (given that the state is the true state of the test subject),and a forwarding address to a remediation program for each state.

A test item pool is a collection of test items. A test item pool isalways linked to a particular poset model. Associated with each testitem in the test item pool are class conditional densities. Theexpression f_(i) (x_(n) |s) denotes the class conditional densityassociated with the n'th administered test item i, x_(n) being one ofthe possible responses to the n'th test item given that the state s isthe true state of the test subject.

The test subject's initial state probability set (SPS) includes a memberfor each state in the poset model and is denoted by π₀. The notation π₀(s) denotes the probability value in the collection of probabilities π₀(s) for the system's prior belief that the test subject belongs to states, where s represents any one of the states. There are a number ofpossible choices for the test subject's initial SPS. One possibility isto assign a non-informative initial SPS which does not take into accountsubjective information about the test subject and thus treats all testsubjects the same. An example of such an initial SPS is a uniform set inwhich all of the probabilities are equal. This choice is attractive inthat there is no need for prior information about the test subject.Another example of a non-informative initial SPS is one in which theprobabilities are derived from the distribution of prior test subjectsamong the poset states.

Ideally, the initial SPS should be tailored to the test subject. Aninitial SPS which heavily weights the true state of the test subjectwill lead to fast and accurate classification. Individualized initialSPSs can be constructed by using prior information concerning the testsubject's domain state. In the case of humans, performance onexaminations, homework, and class recitations can provide guidance. Inthe case of systems, previous operating performance would provide usefulinformation for tailoring an initial SPS.

After the initialization step 3 has been performed, the classificationstep 5 takes place. Information necessary to classify a test subjectinto a domain state is obtained by successively calculating the testsubject's SPS at stage n, denoted by π_(n), after the test subject'sresponse to each of a sequence of N administered test items, n taking onvalues from 1 to N. Additionally, denote the probability at stage n thatthe test subject belongs to state s to be π_(n) (s), for any i in S. Thedetermination of the value of N depends on the stopping rule. Note thatN is random, and that its value is dependent on the current SPS and theremaining available item pool at each stage. The determination of thevalue of N will be described later.

The test subject's posterior probability π.sub.(n+1) (s|X_(n+1) =x,It_(n+1) =i) for membership in state s at stage n+1 is obtained from theequation ##EQU3## where X_(n+1) =x denotes that the test subject'sobserved response at stage n+1 is x, It_(n+1) =i denotes that item i isthe (n+1)th administered item and f_(i) (x|s) is the class conditionaldensity associated with state s evaluated at x for item i. The symbolf_(i) (x|s) denotes a class conditional density associated with either adiscrete or continuous random variable X (see e.g. Steven F. Arnold,MATHEMATICAL STATISTICS, Prentice-Hall, Englewood Cliffs, N.J., 1990,pp. 44-46).

The updating rule represented by the above equation is known as theBayes rule, and it will be the assumed updating rule. Note that itapplies generally when the class conditional density functions are jointdensities and/or conditional densities dependent on previous responses.Other updating rules for obtaining π.sub.(n+1) (s|X_(n+1) =x, It_(n+1)=i) from π_(n) (s) may be used by the system. For alternative updatingrules, it will be assumed that the updated state posterior probabilityvalue be a function of the SPS at stage n and the class conditionaldensities for all the states in S evaluated at observed response x. Theyshould also have the property that for observed response X_(n+1) =x forany item i and fixed conditional density values for all states not equalto s, π.sub.(n+1) (s|X_(n+1) =x, It_(n+1) =i)) is non-decreasing inf_(i) (x|s). This should hold for all s in S and possible responses x.Of course, Bayes rule is an example of such an updating rule.

After N test items have been administered to the test subject, the testsubject is classified. After a test subject is classified, theremediation step 7 can take place by providing the human test subjectwith the knowledge he does not have or by providing a technician thenecessary information to repair at least some of the functional defectsexisting in the test subject.

The flow diagram associated with one embodiment 8 of the classificationstep 5 is shown in FIG. 3. The first step 9 is to clear the test itemcounter which keeps track of the number of test items administered tothe test subject. In step 11, the test item to be administered to thetest subject is selected. A test item is selected from the test itempool by applying an item selection rule.

A useful approach to developing item selection rules is to employ anobjective function to measure the "performance" or "attractiveness" ofan item in the classification process. In practice, this objectivefunction may be dependent upon an item's characteristics such as how itpartitions the poset model, what the corresponding distributions arewithin the partitions, an SPS, and/or the observed item response.Clearly, the probability values in the SPS and the item responses canvary. The objective function can be weighted, usually by a classconditional density, and the weighted objective functionsummed/integrated over all the possible values for the inputs used bythe objective function. In this way, one can obtain an "average" or"expected" value of the objective function which can, for instance,systematically take into account the variation in the SPS and/or thevariation of the possible item responses.

This is done by summing/integrating over all possible input values theproduct of the objective function and the corresponding weightingfunction. Examples are given below. For the examples, it will be assumedthat the system is at stage n, and that the current SPS is π_(n).

An important class of objective functions are uncertainty measures on anSPS. These are defined to be functions on an SPS such that the minimumvalue is attained when all but one of the values in the SPS has valuezero. This minimum may not be unique in that other SPS configurationsmay attain the minimum as well.

A subset of item selection procedures which employ uncertainty measuresas an objective function are those that gauge the uncertainty among themass on an item's partitions with respect to an SPS. For suchprocedures, it is desirable for the item partitions to have a high levelof (weighted) uncertainty. The idea is that the more the mass is spreadacross an item's partitions, the more efficiently the item candiscriminate between states that have significant mass in the SPS ("themore birds that can be killed by one stone"). This is important becausein order to build a dominant posterior probability value in the SPS,item sequences must discriminate between or separate all states withsignificant mass. Conversely, note that if all the mass is on onepartition, there will be no change in the SPS if the updating rule isBayes rule. The motivation of these procedures is to avoid this scenarioas much as possible as measured by an uncertainty measure. Assuming thatall items have a partition corresponding to a PDOI generated by a statein S, consider the simple example below, which selects item i in theavailable item pool that minimizes

    h(π.sub.n, i)=|m.sub.n (i)-0.5|       (4)

and where ##EQU4## and e(i) is the type of test item i. For thiscriterion, as with all others, ties between items can be randomized.Note m_(n) (i) is the mass on one of the partitions of item i at stagen, and the objective function |m_(n) (i)-0.5| measures uncertainty amongthe item partitions with respect to the SPS at stage n, π_(n). Actually,to satisfy the earlier definition of an uncertainty measure, we need tomultiply the objective function by (-1).

This rule is based on a very simple criterion which is an advantage interms of computational complexity. However, the rule is not verysophisticated. It does not take into account the response distributionsof each test item. Also, the rule may not perform well when the testitems have more than two partitions.

Another motivation for classification is that it is desirable for theSPS to have mass concentrated on or around one element. Using theuncertainty measures with the defining property should encourageselection of items that on average lead towards the ideal SPSconfiguration of mass on one point.

An important example of an uncertainty function is Shannon's entropyfunction En(π_(n)) where ##EQU5##

Note that the minimum is indeed attained when one element in the posetmodel has value 1 in the SPS. A weighted version of this objectivecriterion is sh₁ (π_(n),i) where

    sh.sub.1 (π.sub.n, i)=∫En(π.sub.(n+1) |X.sub.(n+1) =x, It.sub.(n+1) =i)P(X.sub.(n+1) =x|π.sub.n, It.sub.(n+1) =i)dx(7)

where i now denotes any test item in the test item pool that has not yetbeen administered to the test subject. The symbol En(π.sub.(n+1)|X.sub.(n+1) =x, It.sub.(n+1) =i) denotes En as a function of πcalculated after the test subject responds to the (n+1)'th administeredtest item given the response by the test subject to the (n+1)'thadministered test item X.sub.(n+1) equals x and the (n+1)'thadministered test item It.sub.(n+1) equals i. The symbol P(X.sub.(n+1)=x|π_(n), It.sub.(n+1) =i) denotes the mixed probability thatX.sub.(n+1) equals x given π_(n) and given that item i was chosen to bethe (n+1)'th administered item.

Note that the equation is based on π.sub.(n+1) |X.sub.(n+1) =x,It.sub.(n+1) =i which denotes the SPS at stage n+1 given the observedresponse for the item administered at stage n+1 is x and the itemselected for stage n+1 is item i. This criterion selects the item i inthe available item pool which minimizes the right-hand side of theequation. Note that the weighting function in this case is P(X.sub.(n+1)=x|π_(n), It.sub.(n+1) =i) which is given by. ##EQU6##

It is a function of the values in the SPS for each state in S multipliedby the density values of the corresponding response distributionsassociated with each state. Indeed, it is a mixed probabilitydistribution on the space of possible response values given It.sub.(n+1)=i and π_(n) and on the poset model S.

If the class conditional density f_(i) (x|s)=f_(i) (x) is associatedwith the response by a state-s test subject to a test item of type e(i)when e(i) is less than or equal to s and f_(i) (x|s)=g_(i) (x) isotherwise associated with the response, then sh₁ (π_(n),i) is given bythe following equation

    sh.sub.1 (π.sub.n,i)=m.sub.n (i)∫En(π.sub.(n+1) |X.sub.(n+1) =x,It.sub.(n+1) =i)f.sub.i (x)dx+(1-m.sub.n (i))∫En(π.sub.(n+1) |X.sub.(n+1) =x, It.sub.(n+1) =i)g.sub.i (x)dx                                          (9)

An alternative to sh₁ (π_(n),i) is sh₁ '(π_(n),i):

    sh.sub.1 '(π.sub.n,i)=sh.sub.1 (π.sub.n,i)-E.sub.n (π.sub.n)(10)

Minimizing sh₁ '(π_(n),i) with respect to i is equivalent to minimizingsh₁ (π_(n),i).

The use of the alternative formulation sh' can lead to a reduction incomputational complexity since in the two-partition case, it can beviewed as a convex function of m_(n) (i). Employing computationallysimple item selection rules aids in the feasibility of employing largeposet models and employing k-step extensions (see below).

A generalization of this class of selection rules in one that selects atest item to be administered to a test subject which minimizes theexpected value of an SPS function after taking into account the possibleresponses to the next k administered test items, k being an integer.Item selection rules which look ahead k steps are attractive in thatthey are better able to exploit the potential of the items remaining inthe test item pool.

The expected value sh_(k) (π_(n),i) of En(π_(n)) after administering ktest items can be calculated in a straightforward manner using therecursive formula ##EQU7## where "min over j" means the value of thequantity in brackets for an item j from the preceding available itempool which minimizes the value. The version of the equation where P canbe represented by f_(i) and g_(i) is ##EQU8## where e(i) is the type oftest item i.

The same framework for constructing item selection rules applies todistance measures on two different SPSs: for instance, π_(n) andπ.sub.(n+1) |X.sub.(n+1) =x, It.sub.(n+1) =i. Let a distance measurebetween two SPSs be such that, given SPSs a and b, the distance measureis a function of a and b that attains its minimum given a when a=b. Notethat this minimum may not necessarily be unique. The motivation behindsuch a measure is that it is undesirable for an item not to lead tochange between successive SPSs. An example of such a distance functionis the sum over all of the states of the absolute difference ofcorresponding SPS elements associated with each state. Consider the itemselection rule based on this objective function which selects item i inthe available item pool that maximizes Kg(π_(n),i) where ##EQU9##

The version of this equation that is obtained when item i has twopartitions represented by f_(i) and g_(i) and is associated with typee(i) is ##EQU10##

Note that each term in the sum comprising the distance function on theSPSs is weighted correspondingly by the weighting function π_(n)(s)f_(i) (x|s) for each s in S and possible response x givenIt.sub.(n+1) =i and π_(n). ##EQU11## where "min over j" means the valueof the quantity in brackets for an item j from the preceding availableitem pool which minimizes the value.

Yet another important class of item selection rules are based onobjective functions that measure the "distance" or "discrepancy" betweenclass conditional densities associated with the various states in S. Theterm "distance" or "discrepancy" is to be interpreted as a measure ofthe discrimination between the class conditional densities. Formally, itis assumed that a discrepancy measure is a function of two classconditional densities such that, for class conditional densities c andd, the discrepancy measure takes on its minimum given c when c=d. Thisminimum may not be unique. The motivation of adopting such objectivefunctions is that items become more desirable for classification as thediscrepancy between its class conditional densities increases.Conversely, if class conditional densities are equivalent, thenstatistically there will be no relative discrimination between therespective states in the subsequent SPS.

An example of item selection rules based on such objective functionsinclude those that select the item i in the available item pool whichmaximizes the weighted discrepancies wd(π_(n),i) where ##EQU12## whered_(jk) (i) is a discrepancy measure between the class conditionaldensities of states j and k for item i. Note that each distance betweena pair of states is weighted by the product of the correspondingprobability values in the current SPS. A particularly simple d_(jk) (i)is the one which equals 0 if f_(i) (x|j) equals f_(i) (x|k) and 1otherwise.

As an illustration, suppose item i partitions the set of states into twosubsets with item type denoted by e(i). Suppose f_(i) (x|j) equals f_(i)when e(i)≦j and equals g_(i) otherwise. Examples of discrepancy measuresfor f_(i) and g_(i) include the Kullback-Liebler distance given by

    d.sub.jk (i)=∫log f.sub.i (x)/g.sub.i (x)!f.sub.i (x)dx; j≧e(i), k≧/e(i)

    d.sub.jk (i)=∫log g.sub.i (x)/f.sub.i (x)!g.sub.i (x)dx; j≧/e(i), k≧e(i)                             (17)

    d.sub.jk (i)=0j,k≧e(i), j,k≧/e(i)

and the Hellinger distance given by ##EQU13##

Still another class of item selection rules are the k-step look-aheadrules. These rules employ as objective functions loss functions such asthose described earlier. Again, the objective functions will usually beweighted over the possible input values. The motivation behind suchcriteria is to reduce the average cost of misclassification whilebalancing the average cost of observation. There are a variety ofpossible loss functions that one might use. Importantly, the lossfunction used in item selection may differ from that used in theintegrated risk determination (see above). If the same loss function isused, then the k-step look-ahead rule selects the best k-step strategywhich leads to the greatest reduction in the integrated risk within ak-step horizon. Note that it is possible that less than k items may beadministered in a k-step strategy.

A one-step look-ahead rule can be based on the expected loss LA₁ definedby the equation ##EQU14## where L(s,d(x),1) is the loss function, anditem i is selected from the available test item pool. Of the remainingyet to be administered items in the test item pool, the one which isassociated with the smallest value of LA₁ would be chosen as the(n+1)'th item to be administered. It may be assumed that d(x) is theBayes decision rule after response x is observed.

If the class conditional density f_(i) (x|s)=f_(i) (x) is associatedwith the response by a state-s test subject to a test item i of typee(i) when e(i) is less than or equal to s and f₁ (x|s)=g_(i) (x) isotherwise associated with the response, then LA₁ (π_(n),i) is given bythe following equation ##EQU15## where L(s,d(x),1) can be viewed as afunction of π.sub.(n+1) |X.sub.(n+1) =x, It.sub.(n+1) =i. If the lossfunction has constant cost of observation and 0-1 misclassificationcost, this criterion reduces to choosing the item that will give thelargest expected posterior value in π.sub.(n+1).

A k-step look-ahead rule utilizes the expected loss LA_(k) inadministering the next k test items. The quantity LA_(k) is definedrecursively by the equation ##EQU16## where "min over j" means the valueof the quantity in brackets for an item j from the preceding availableitem pool which minimizes the value. The version of the equation whenitem i has two partitions represented by f_(i) and g_(i) and isassociated with type e(i) is ##EQU17##

Not all reasonable item selection rules need be based directly onobjective functions. First, let us begin with the definition of animportant concept in item selection. An item i is said to separate thestates s₁ and s₂ in S if the integral/sum over all possible responses ofthe class conditional density f_(i) given s₁ and/or f_(i) given s₂ ofthe absolute difference of the class conditional densities is greaterthan zero. In other words, states s₁ and s₂ are separated if, withpositive probability with respect to one of the densities, therespective two-class conditional densities are different. Thisdefinition can be generalized to: an item is said to separate two statesif for a discrepancy measure such as in equations (18) or (19) for thecorresponding class conditional densities, the resultant value exceeds apredetermined value. The class of discrepancy measures utilized in theinvention coincides with those utilized in item selection rules based onweighted discrepancy measures. Indeed, the criterion for separation canbe generalized further by considering a plurality of discrepancymeasures, and establishing the separation criterion to be satisfied iffor instance two or more measures exceed predetermined values, or allthe measures exceed a predetermined value, or other such conditions.

Let us now introduce the function Φ which, given a separation criterionand two states in S, determines if an item from a given item pool indeedseparates the two states. The outcome "yes" can be assigned the value 1and "no" the value 0. An application of Φ is to generate for two statesthe subset of items which separates them from the available item pool.

As an illustration, suppose the poset in FIG. 6 is the underlying modelS. Further, let the corresponding item pool contain four items, eachwith two partitions, of types C, AC, BC, and 1 respectively and whosecorresponding class conditional densities satisfy the given property ofseparation. Then, for states AC and BC for instance, given this itempool, Φ can be used to generate the subset of items which separate them,the items of types AC and BC. All this involves is to group together allitems for which their Φ-values are equal to 1.

The function Φ can also be used to generate reasonable item selectionrules. One procedure is as follows:

1. Find the two states in S with the largest values in the current SPSat stage n;

2. Use Φ to identify items in the available item pool that will separatethese states;

3. Select the item in the resultant subset of items provided by Φ withthe largest discrepancy value with respect to a discrepancy measure suchas in equations (18) and (19) of the class conditional densities of thetwo states, or, allow the system to randomize selection among thoseitems, thus avoiding use of an objective function altogether. The classof discrepancy measures that can be used in this item selectionprocedure is equivalent to the class that can be used in the itemselection rules based on discrepancy measures on class conditionaldensities.

All the rules discussed above can be randomized. This involvesintroducing the possibility that the item selected by a given rule at agiven stage may, with positive probability be exchanged for anotheritem. This randomization may be weighted by the relative attractivenessof the items with respect to the item selection criterion.

One technique which implements such randomization of item selection issimulated annealing (see S. Kirkpatrick et al., "Optimization bySimulated Annealing", Science, 220, pp. 671-679). The inclination to"jump" to another item is regulated by a "temperature" (i.e. theprobability distribution associated with the randomization process iscontrolled by a "temperature" parameter). The higher the temperature,the more likely a jump will occur. An item selection rule used inconjunction with simulated annealing can be run at various temperatures,with each run referred to as an annealing.

Specifically, one implementation of simulated annealing would be toregulate jumping with a Bernoulli trial, with the probability of jumpinga function of the temperature parameter. The higher the temperature, thehigher the probability that a jump will indeed occur. Once a jump hasbeen decided upon, the probability distribution associated withalternative items could for instance be proportional to the respectiverelative attractiveness of the items with respect to the item selectioncriterion in question.

The motivation for employing such a modification to an item selectionrule is that sometimes myopic optimization may not necessarily lead toitem sequences with good overall performance as measured for instance bythe integrated risk. Several annealings can be run and the correspondingstrategies analyzed to see if improvement is possible.

Another technique for possibly improving upon a collection of itemselection rules is to hybridize them within a k-step horizon. Thisprocedure develops new item selection rules based upon collections ofother rules. For each rule in a given collection, a k-step strategy isconstructed at each stage in the classification process. The hybridizedrule selects the item which was used initially in the best of the k-stepstrategies as judged by a criterion such as the integrated risk withrespect to the current state SPS. (A different loss function to judgethe k-step strategies than the one used for the general classificationprocess may be used.) Hence, the hybridized rule employs the itemselection rule which is "best" at each particular state in terms of ak-step horizon, so that overall performance should be improved over justusing one item selection procedure alone.

Other hybridizing techniques are possible. As an example, given aplurality of item selection rules, an item can be selected randomly fromthe selections of the rules. Alternatively, each test item in theavailable test item pool can be assigned a relative ranking ofattractiveness with respect to each selection rule: for instance "1" forthe most attractive, "2" for the second most attractive, etc. The testitem with the highest average ranking among the selection rules isselected. Clearly, the ranking values can also be based on the relativevalues of weighted objective functions. In general, criteria based onweighted relative rankings of attractiveness with respect to a pluralityof item selection rules will be referred to as relative rankingmeasures, with the higher the weighted relative ranking, the moreattractive the item.

After selecting the next test item in step 11 of the classificationprocess 5 shown in FIG. 3, the selected test item is flagged whichindicates that the selected test item is not available for futureselection. The selected test item is then administered to the testsubject and the test subject's response is recorded in step 13. The testsubject's SPS is then updated in step 15 in accordance with equation (3)and the test item counter is incremented in step 17.

The decision is made in step 19 as to whether at this point theadministering of test items should be stopped and the test subjectclassified. The simplest criterion for making this decision is whetheror not any of the members of the test subject's SPS exceeds aclassification threshold. If any of the members of the test subject'sSPS does exceed the threshold, the test subject is classified in step 21in the state associated with the member of the test subject's SPS havingthe greatest value and the remediation process 7 begins.

If none of the members of the test subject's SPS exceeds theclassification threshold, the test item count recorded in the test itemcounter is compared with a test item limit in step 23. If the test itemcount is less than the test item limit, the classification processreturns to the item selection step 11 and the process continues. If thetest item count exceeds the test item limit, it is concluded that theclassification process is not succeeding and the classification processis terminated in step 25. The classification process may not succeed fora variety of reasons. For example, if the responses provided by the testsubject are inconsistent with respect to any state in S (i.e. nodominant posterior probability emerges in the SPS functions), it wouldbe impossible to properly classify the test subject.

Another possible stopping rule that may be employed is the k-steplook-ahead stopping rule. It involves the same calculations as with ak-step look-ahead item selection rule and results in a k-step strategyδ_(k) with respect to the classification decision-theoretic lossfunction.

Given a current SPS, the system must decide whether to continue or stop.The k-step look-ahead stopping rule will favor stopping if R(π_(n),δ_(k))>=R(π_(n), δ₀), where δ₀ is the strategy that stops at the currentposition. The strategy δ_(k) may be represented by a strategy tree (seebelow). Of course, other item selection criteria can be used toconstruct δ_(k) besides that of the equation for LA_(k) given above.Additionally, the loss function used in the k-step look-ahead stoppingcriterion may differ from those used in other contexts.

The k-step look-ahead stopping rules can be based on other weightedobjective criteria besides a loss function. Consider the uncertainty anddistance measures on SPS vectors. After constructing δ_(k) at a givenstage, if the weighted (expected) reduction in an uncertainty measure isless than a predetermined value, or the increase in the distance betweenthe weighted (expected) SPS at stage n+k and the current SPS is notgreater than a specified value, stopping may be invoked.

Stopping rules do not necessarily have to look ahead k steps. A stoppingrule may be a function of the current SPS. For instance, if a weighteduncertainty measure on the current SPS is less than a predeterminedvalue, stopping can be invoked. Similarly, if a weighted distancemeasure, for instance, between the initial SPS and the current one islarger than a predetermined value, it would be attractive to stop, andstopping can be called. Using loss functions, a stopping rule coulddepend on whether or not a weighted loss is less than a predeterminedvalue. A stopping rule could be based on such a criterion as well.Weighting for these stopping rule criteria could for instance be withrespect to the class conditional density values corresponding to thetest item responses administered up to the current stage and the initialSPS.

Consider the following examples. Suppose a loss function has a cost ofobservation of 0 until n>10 and then becomes 1 with no misclassificationcost. The corresponding stopping rule for this loss function will invokestopping if and only if the number of observations reaches 10 (cf. FIG.3, step 23). Note how this loss function belongs to the class of lossfunctions described earlier. Also note that this loss function istailored for developing reasonable stopping rules and may not correspondto the loss function used in the integrated risk function.

Consider now the uncertainty measure which calculates the quantity (1minus the largest posterior probability in the SPS). The correspondingstopping rule could then be the simple one described above, which stopsif the largest posterior probability in the current SPS exceeds athreshold value. Note that the two examples described above can be usedin conjunction to develop a stopping criterion, such as invokingstopping if and only if one or both of the rules calls for stopping. Analternative would be to invoke stopping if and only if both rules callfor stopping. Clearly, with a plurality of stopping rules, various suchcombinations can be used in constructing a new stopping rule.

Recall that the decision rule which minimizes the integrated risk withrespect to a loss function and initial SPS is called the Bayes decisionrule. The decision rule is a function of the observed response path andits corresponding response distributions. Due to computationaldifficulty, it may sometimes be easier to use a Bayes decision rule froma different context (i.e. different initial SPS and different lossfunction). For example, if misclassification costs vary among the statesin S, it may not always be the Bayes decision rule to select the statewith the largest posterior probability in the final SPS, yet it maystill be attractive to do so.

Moreover, when the underlying poset model has an infinite number ofstates, it is possible for purposes of deriving a decision rule to letthe initial SPS have infinite mass. The best decision rules in terms ofminimizing the integrated risk with respect to such initial SPS priordistributions are called generalized Bayes rules. These rules also maybe useful. Once again, note that the loss functions used in the decisionprocess may differ from those used in the classification process(integrated risk criterion) and those used in item selection and/orstopping. As in item selection, when using stopping or classificationdecision criteria, ties between decisions can be randomized. Foremphasis, it should be noted that item selection and/or stopping rulescan vary from stage to stage and decision rules from test subject totest subject.

A portion of a strategy tree embodiment 31 of the classification step 5is shown in FIG. 4. A strategy tree specifies the first test item to beadministered together with all subsequent test items to be administered.Each test item in the strategy tree after the first is based on the testsubject's response to the last test item administered and the updatedSPS. Strategy trees are representations of strategies. A strategy treeis a plurality of paths, each path beginning with the first test item tobe administered, continuing through a sequence alternating between aparticular response to the last test item and the specification of thenext test item, and ending with a particular response to the final testitem in the path. The classification of the test subject, based on thefinal updated SPS for each path, is specified for each path of thestrategy tree. Note that strategy trees can be used when the responsedistributions are continuous if there are a finite number of possibleresponse intervals associated with an item choice. Also, multiplebranches emanating from a node in the tree indicates multiple possibleresponse outcomes.

Thus, the identity of the next test item in the strategy tree can bedetermined by referencing a memory location keyed to the identity of thelast test item administered and the response given by the test subjectto the last test item. The last test item to be administered for eachpath in the strategy tree is identified as such in the memory, and eachresponse to that last test item is associated in memory with theappropriate classification of the test subject who has followed the paththat includes that particular response. Directions for remediation arealso stored in memory for each path of the strategy tree.

It is assumed in FIG. 4 that the test subject's response to a test itemcan be either positive or negative. The first item to be administered isspecified by the strategy tree to be item-3 and is administered in step33. The response is analyzed in step 35. If the response to item-3 ispositive, the next item to be administered is specified by the strategytree to be item-4 which is administered in step 37. The response toitem-4 is analyzed in step 39. If the response to item-4 is positive,the administering of test items ceases, classification of the testsubject occurs in step 40, and the test subject transitions to theremediation step 7 (FIG. 2). If the response to item-4 is negative,item-7 is administered in step 41 in accordance with the strategy treespecification. The process continues in a similar manner after step 41until a stopping point is reached and classification occurs.

If the response to item-3 is determined in step 35 to be negative,item-1 is administered in step 43 as specified by the strategy tree andanalyzed in step 45. If the response to item-1 is positive, theadministering of test items ceases, classification of the test subjectoccurs in step 46, and the test subject transitions to the remediationstep 7 (FIG. 2). If the response to item-1 is negative, either item-9,item-2, . . . , item-5 is administered in steps 47, 49, . . . , 51, asspecified by the strategy tree. The process continues after these stepsuntil a stopping point is reached and classification occurs.

Such strategy trees are developed starting with the initial SPS and thetest item pool, and determining the sequence of test items to beadministered using the test item selection procedures described above. Astrategy tree branches with each administration of a test item untilstopping is invoked by a stopping rule.

It may be possible to create a more efficient strategy tree from anexisting one by evaluating a weighted loss function one or more testitems back from the final test item in a path and determining whetherthe additional test items in the strategy tree are justified by areduction in the weighted loss function.

The relationship between the loss function and a strategy tree isillustrated in FIG. 5. A circle 61 denotes a test item and line segments63 and 65 denote the possible responses to the test item. Test item 67,the first test item in the strategy tree, is the beginning of all pathsin the strategy tree. Each path terminates with a line segment such asline segment 63 which does not connect to another test item. The lossfunction L(s,d,n) for each path can be determined after classificationoccurs at the end of each path, assuming the test subject's trueclassification is s, as indicated in FIG. 5.

The loss function cannot be used directly in refining a strategy treesince one never knows with absolute certainty the true classification ofa test subject. Instead, the weighted loss function (i.e. integratedrisk) R(π₀, δ) is used for this purpose.

As mentioned above, a strategy tree can be refined by using the weightedloss function. Suppose the weighted loss function of the strategy treeδ₁ of FIG. 5 is R(π₀, δ₁). Now eliminate test item 62 and call thisrevised tree δ₂ with an weighted loss function R(π₀, δ₂). If R(π₀, δ₂)is less than R(π₀, δ₁), the reduced weighted loss function suggests thatstrategy tree δ₂ is preferable to original strategy tree δ₁.

Rather than eliminating only one test item, one might choose toeliminate test items 62 and 64, thereby obtaining strategy tree δ₃.Again, if R(π₀, δ₃) is less than R(π₀, δ₁), the reduced weighted lossfunction suggests that strategy tree δ₃ is preferable to originalstrategy tree δ₁. There are obviously many possible modifications of theoriginal strategy tree that might be investigated using the weightedloss function as the criterion of goodness. A systematic approach wouldbe to employ a "peel-back" approach. This entails "growing" the treewith a computationally-simple stopping rule such as the one whichdecides to stop when one of the states in S has a posterior probabilityvalue which exceeds a threshold value or when the number of observationsexceeds a threshold. Then, the system can "peel-back" the tree andrefine the stopping rule in terms of the weighted loss function byapplying a k-step look-ahead stopping rule only to all the sub-trees atthe end of the tree with a branch at most k steps from termination(k>=1). This approach becomes attractive when applying the k-steplook-ahead stopping rule at each stage in the strategy tree iscomputationally expensive.

An important application of the technology used to generate sequentialtest sequences is in the development of fixed sequence tests. A fixedsequence test (fixed test) is a sequence of items that are to beadministered to all test subjects, with no sequential selectioninvolved. A test length may be predetermined or can be determined duringdesign given a decision-theoretic framework as used in the sequentialsetting. Indeed, the same classification framework can be used in thefixed test context as well (use of loss functions with costs ofmisclassification and observation, integrated risk functions, an initialSPS, etc.). The objective for this problem then is to choose the fixedsequence from an item pool which minimizes the integrated risk for agiven loss function and initial SPS. Note that choosing the test length(i.e. deciding when to stop) may be an issue since the loss function mayinclude a cost of observation. Also, note that during actualadministration of a given fixed test, it is possible to allow testsubjects to stop before completing all of the test items in the fixedsequence, using stopping rules as described earlier. Decision rules areanalogous in the fixed test context in that their objective is to make aclassification decision which minimizes a weighted loss function.

All the previous item selection rules such as those based on weightedobjective functions can be adapted to this application as well, alongwith the techniques of extending them for k-step horizons, hybridizing acollection of them, and introducing randomization to the selectionprocess. As an example, items can be selected iteratively via thesh-criterion by choosing at stage n+1 the item i from the remainingavailable item pool which minimizes ##EQU18## where i₁, i₁, . . . ,i_(n) are the previously selected items at stage 1 up through stage nrespectively and ##EQU19##

The function f is the joint class conditional density for responses x₁,. . . , x_(n), x_(n+1) given state s and item sequence i₁, . . . ,i_(n), i. In addition, the probability of a test subject being in aparticular test item partition can be calculated for instance byweighting the probability values that would be given by the possibleSPSs that could result from administration of the fixed test items up tostate n. Recall that the probabilities of a test subject being in a testitem's partitions are quantities used by certain item selection rules.

Item selection criteria based on the function Φ can also be used in thiscontext as well. First, list all pairs of states that need to beseparated, optionally giving more weight to certain separations (e.g.requiring that a certain separation should be done twice). The objectivein selecting a fixed sequence would then be to conduct as many of thedesired separations as possible, using for a given pair of states and agiven separation criterion the function Φ to determine whether an itemresults in a separation between them. An item selection criterion wouldbe to choose an item which results in as many of the remaining desiredseparations as possible. Once an item is administered, the list ofdesired remaining separations is updated by removing the resultantseparations.

In the strategy tree context, the restriction that the same itemsequence be administered to all test subjects is equivalent to requiringall branches in a tree to be equivalent. In general, one can view theprocess of selecting a fixed test as a special case of the generalsequential analytic problem. At each stage n of the tree-buildingprocess, n>=1, instead of allowing each node to be associated with itsown item, developing a fixed test is equivalent to requiring that allnodes at the same stage n of the test share the same item selection.Note that the "peel-back" approach to constructing a stopping rule canstill be applied.

Conversely, developing fixed test sequences has application insequential testing. Recall k-step look-ahead item selection and stoppingrules, which require development of a k-step horizon strategy at eachstage. This can be computationally costly if k is large and the posetmodel and item pool are complex. As an alternative, one can insteadcalculate a fixed test sequence within a k-step horizon in place of ak-step strategy. Criteria for item selection and stopping based on usinga k-step horizon fixed test are analogous.

For both the sequential and fixed test settings, the above techniquescan be used to design the item pool (fixed test) in terms of what typeof items should be constructed. To gain insight, classification isconducted on hypothetical items with hypothetical item types and itemresponse distributions. Since the classification process is beingsimulated, an infinite number of each of the item types of interestwithin a range of class conditional densities that reflect what is to beexpected in practice can be assumed. From the hypothetical item pool,strategy trees or fixed sequences can be constructed for various initialSPS configurations. The composition of these constructions in terms ofthe hypothetical item types selected gives guidance as to how to developthe actual item pool or fixed sequence. Hypothetical item types thatappear most frequently on average and/or have high probability ofadministration for instance with respect to SPSs and class conditionaldensities are candidates to be constructed. Analyzing the itemcomposition of a number of simulated classification processes is analternative approach to gaining insight into item pool design. Note thatthese approaches can be applied to actual test item pools as well.Actual test items that are not administered with high frequency onaverage and/or do not have high probability of administration, forinstance with respect to SPSs and class conditional densities, arecandidates for removal.

An important consideration in the implementation of the invention is thedevelopment of a model of the domain of interest and the associated testitem pool. Concerns in developing the model include whether the modelhas too many or too few states. Desirable properties of the test itempool include having accurately specified items which stronglydiscriminate between states and having a sufficient assortment of itemtypes to allow for effective partitioning of the states.

A model is too large when some of the states are superfluous and can beremoved without adversely affecting classification performance. A modelis too small when some of the important states are missing. An advantageto having a parsimonious model is that for test subjects in states thatare specified, it doesn't require on average as many test items to reachthe classification stage and to classify with a particular probabilityof error as it does for a larger model which contains the smaller one.The disadvantage is that test subjects in states that are not present inthe model cannot be appropriately classified.

A good model gives useful information concerning the remediation of testsubjects. Each state should be meaningful in assessing theknowledgeability or functionality of the test subject. Moreover, themodel should be complex enough to be a good representation of all therelevant knowledge or functionality states in a given subject domain.Hence, balancing parsimony while accurately representing the subjectdomain is the primary challenge of model development.

The selection of items for the test item pool entails determining howeffective a test item is in distinguishing between subsets of states.The effectiveness of a test item is determined by the degree ofdiscrimination provided by the response distributions associated withthe test item and the subsets of states. The degree of discriminationprovided by response distributions can be measured in a variety of ways.Two possibilities are illustrated by equations (18) and (19), withlarger values indicating a larger degree of discrimination. In general,discrepancy measures from the same class as employed in item selectioncan be used.

The starting point for the development of a domain model and itsassociated test item pool is the postulating of model candidates byexperts in the field of the domain and the generation of test itemcandidates of specified types for the test item pool. Within each model,the experts may have an idea as to which states may be superfluous andwhere there may be missing states. Further, the experts may have an ideaas to which items do not discriminate well between subsets of states orwhose association with the domain states may be vague and need to beinvestigated. These prior suspicions are helpful in that they allow theuser to experiment through design of a training sample of test subjectsin order to gain information necessary to make decisions about itemperformance and model structure.

With respect to the relationship between domain models and the test itempool, it is of considerable importance that the item pool candiscriminate among all of the states. Whether this is true or not can bedetermined by a mapping on the poset model, given a test item pool withfixed item partitions. In general, the item partitions may be specifiedsuch that they do not necessarily correspond to the subsets with sharedclass conditional response distributions, and in fact can be specifiedwithout taking into consideration actual estimated class conditionalresponse densities. Moreover, separation criteria can be used forspecifying alternative partitions, such as grouping together stateswhose class conditional density discrepancies are small. Thesealternative partitions can be used below and in item selection rules.The mapping consists of the following sequence of operations:partitioning the domain set of states by means of a first item in thetest item pool into its corresponding partition, partitioning each ofthe subsequent subsets in the same manner by means of a second item,resulting in the further possible partitioning of each partition of thefirst item; continuing the partitioning of the resultant subsets at eachstage of this process by means of a third, fourth, . . . , nth type ofitem until either there are no more items left in the item pool or untileach state in the original poset is by itself in a subset. The lattersituation implies that the item pool can discriminate between all of thestates in the domain in relation to the fixed item partitions. If thefinal collection of subsets contains one subset that has more than onemember, the implication is that the test item pool cannot discriminatebetween those states in that subset, again in relation to the fixed itempartitions. The image of this mapping can be partially ordered, with thepartial order induced in the sense that x'≦y' for x' and y' in the imageif there exists x and y in the original poset such that x≦y and theimages of x and y are x' and y' respectively.

FIGS. 6 and 7 give an illustration of this mapping. Suppose the itempool contains 4 items, each with two partitions and associatedrespectively with states {C,AC,BC,1}. FIG. 6 shows the original posetmodel. FIG. 7 is the image of the mapping on the poset model of FIG. 6.The image shown in FIG. 6 indicates that the test item pool was unableto separate states 0, A, B, and AB.

A resultant image poset can be viewed as the effective working model forclassification in relation to the item pool and given item partitionsand is a reduction from the original poset model. In practice, thisreduction in the number of states can be substantial, depending on theitem pool. States that are not discriminated by the mapping areeffectively viewed as one state in the image. Also, if classification isto be conducted on the image poset, note that an item's partition mustbe updated in relation to the new model. If the partial order on theimage is induced as above, then an item's partition in the image is justthe image of the partition, and the system can automatically update theitem type specification.

The unavailability of item types is a natural constraint for the model.Certain item types may be awkward, such as an item requiring exactly oneskill. Item construction constraints are a factor in the type of modelsthat can be used in classification. Thus, the mapping described abovegives important information about the type of models that can beconstructed and whether the item pool needs to be augmented in order tobetter separate states. It can be used to analyze the performance of afixed test, to see which states may not be separated by the fixed test.

The mapping on the domain model should be performed immediately aftercandidate domain models and candidate test item pools have been definedin order to provide insight as to possible constraints imposed on themodel and possible flaws in the process for selecting candidate itemsfor the test item pool. The mapping should be repeated for anymodifications of the candidate models and test item pools.

Sometimes it is of interest to generate ideal response patterns.Consider the following example from the educational application. Given aposet model, suppose that the response distributions for the items areBernoulli, and that each item has two partitions. Then, given each item,it can be determined whether a test subject in a specified state in theposet model has the knowledge or functionality to give a positiveresponse, in which case a "1" is used to denote the response. Otherwisea "0" is used to denote the response. The final sequence of 1s and 0s,ordered in the same way as the test items, is referred to as the idealresponse pattern. For this case, the ideal response pattern for a testsubject in the specified state are the responses that would be observedif the test subject's responses perfectly reflected the test subject'sstate.

In general, an ideal response for an item given a specified state can beany representative value of the class conditional density for the item.In the continuous response case, this could be the mean of the density.Further, instead of an ideal response value, the ideal response can berepresented by an ideal set of values possibly including ideal intervalsof values (depending on whether the class conditional density isdiscrete or continuous). An example of a possibly useful ideal responseinterval for an item given a specified state is the set of values withina specified distance from the class conditional density mean for thatstate. When the response is multi-dimensional, an ideal response couldbe a value or set of values in the multi-dimensional space of possibleresponses. Ideal response patterns will contain information about eachitem in a given item sequence.

Given a test subject response pattern g and an ideal pattern h, distancemeasures on the patterns can be used to gauge whether the ideal patternis "close" to the test subject pattern. For the example above, areasonable distance measure would be to count the number ofdiscrepancies between patterns, with an ideal pattern said to be "close"to a test subject pattern if that number is less than a certainspecified number, or, equivalently, if the percentage of discrepanciesis less than a specified percentage. In general, we will considerdistance measures between a test subject pattern g and an ideal patternh given an administered sequence of test items such that, given a testsubject pattern g, a distance measure will attain its minimum when g=h,where g is said to equal h, when h has ideal responses that are a valueor set of values, when test subject responses are equal to or containedwithin the corresponding ideal responses. This minimum may not beunique.

These ideal response patterns can be used in model development in thefollowing way. Given an exploratory sample of test subject responsepatterns, a state associated with an ideal pattern that is not "close"with respect to a given distance measure to any test subject's actualresponse pattern suggests that the state may be illusory and should beremoved from the domain model. Conversely, if there are a number of testsubject response patterns not "close" to any ideal patterns, thissuggests that more states may need to be specified. Note that each testsubject may have his own sequence of items, which would entailgenerating ideal responses for each state for each sequence. Also, aswith combining a plurality of stopping rules to develop a new stoppingrule, a plurality of distance measures can be combined to develop adistance criterion.

An essential step in the test subject classification process is thedetermination of the parameter values that characterize the classconditional densities associated with each test item and with each testsubject state. Bayesian estimation of the class conditional densityparameters is a standard statistical approach for obtaining parameterestimates and is analogous to the decision-theoretic framework used inclassification (see Steven F. Arnold, MATHEMATICAL STATISTICS,Prentice-Hall, Englewood Cliffs, N.J., 1990, pp. 535-570). Bayesianestimation treats an unobserved parameter (such as the probability thata test subject in a given state will provide a positive response to atest item) as a random variable. The first step is the selection of aninitial marginal density for the parameter of interest. This initialmarginal density, called the prior distribution, represents a priorbelief about the value of the parameter. Data is then collected and theprior distribution is updated using the Bayes rule to obtain a posteriordistribution. A decision rule is employed that analyzes the posteriordistribution and determines an estimate of the parameter. The "best"decision rule, provided it exists, is called the Bayes decision rule ifit minimizes the integrated risk. The integrated risk is the expectedvalue of a risk function with respect to the prior distribution, therisk function being the expected value of a loss function, and the lossfunction being a measure of the discrepancies between an estimate andthe true parameter value. If the loss function is squared error loss fora continuous parameter, then the penalty in inaccuracy is measured bysquaring the discrepancy between an estimate and the true value. Forthis particular loss function the Bayes decision rule is to take as theestimate the mean of the posterior distribution.

Bayesian estimation is just one way of obtaining parameter estimates.There are many other viable approaches to parameter estimation that donot necessarily involve specifying prior distributions, loss functions,etc.

It may be appropriate, depending on the situation, to perform trialclassifications of a sample of test subjects using the candidate domainmodels and candidate test items. Such preliminary trials may revealproblems with the domain model (i.e. too many or too few states),problems with specifying item type, and problems with itemdiscrimination between subsets of states. The preliminary trials mayalso be helpful in specifying the response distribution parameters forthe test items and the initial SPS. A mapping of how well the items inthe test item pool discriminate among the states of the domain model maysuggest additional types of test items. Finally, a loss function for usein generating strategy trees must be specified.

Further development of the domain model and test item pool depends ontests of this initial configuration using a training sample of testsubjects. The focus of the training sample experiments is to collectdata concerning model fit and item effectiveness and to do so asefficiently as possible. Item selection for the training sample can beaccomplished sequentially as previously described. For this purpose, atraining sample strategy tree can be generated. The initialconfiguration is then exercised with the training sample.

Item types or states that are of particular concern should be thesubject of replicated observations. For items of questionable utilityinsofar as discriminating among states, the updating of the SPS can bedeferred so that the observation conditions for a sequence of test itemsremains the same. Sequences conditional on specific response patternscan be inserted into the item selection sequence/strategy tree and usedto test the presence of hidden states or to see if a state issuperfluously included.

It is important that every parameter be estimated with enough data toinsure accuracy. Items that might not be administered very often shouldpossibly be inserted into the item sequence/strategy tree. Moreover,classification can be delayed by administering more test items to insureclassification accuracy.

Data resulting from tests of the training sample are used to refine theestimates of response distribution parameters, and adjustments are madein the model and in the test item pool. Such adjustments may includerespecifying item types and condensing models. The adjustedconfiguration is then exercised, possibly resulting in furtheradjustments in the domain model and the test item pool.

An estimate of the average classification performance expressed as theweighted value of the loss function in relation to various values of theinitial SPS is then obtained through simulation based on the estimatedresults from the test item pool. This information is useful in decidingupon a final model and test item pool that will be used in classifyingfuture test subjects.

Using the initial estimated item parameter values as specified by theprior distributions, classification of the test subjects in the trainingsample can be accomplished. A classifying function of the test subject'sSPS is used to classify the test subjects. Based on theseclassifications, the item parameter values are updated. Theclassifications can then be updated using the new parameter estimates.This iterative process continues until there is reasonable confidencethat estimation has been accomplished correctly, such as having theestimated values from iteration to iteration appearing to converge, e.g.the estimated values from iteration to iteration after a certain stageare within a predetermined difference.

An approach to choosing a test subject's classification is one whichrandomly selects a state in accordance with the current SPS values. Oneexample of this approach is the estimation technique known as Gibbssampling which utilizes an iterative framework (cf. A. Gelfand and A.Smith, "Sampling-Based Approaches to Calculating Marginal Densities,Journal of the American Statistical Association 85, 398-409 (1990)). Forthe system, it will many times be of interest to impose orderconstraints among the parameters associated with an item's partitions.Estimation in this context can also be conducted via Gibbs sampling (cf.A Gelfand, A. Smith, and T. M. Lee, "Bayesian Analysis of ConstrainedParameter and Truncated Data Problems Using Gibbs Sampling", Journal ofthe American Statistical Association, 87, 523-532 (1992)).

Classification is very important to item parameter estimation because itis used as the basis of the estimates. More precisely, theclassification information is used to represent test subjects' statemembership. Estimation of the item parameters would be simple if thetrue states were known. The difficulty arises precisely because the testsubjects' states are unknown. The sequential process of administeringtest items to the training sample of test subjects results in "sharp"posterior probabilities for state membership near 1 or 0, which in turnresults in "sharper" item parameter estimates. Discrepancy measures suchas those defined by equations (18) and (19) can be used to gaugesharpness. As in developing a separation criterion, a plurality of thesemeasures can be combined for a sharpness criterion. For Bernoulli ormultinomial distributions, "sharp" probabilities are those near 1 or 0with one dominant response for each state.

After item parameter estimation for the training sample of test subjectshas been completed, items that are not sharp are removed. Such itemshave a relatively greater likelihood of giving contradictory evidence asto the identity of the true state because they do not differentiatebetween states very well.

A measure of an item's effectiveness is how the performance ofclassification changes overall when it is removed from the item pool.Responses to items that are not sharp have a relatively higherlikelihood of giving contradictory information which decreases thesharpness of the posterior values of the SPS and is expensive in termsof the cost of observation needed to rectify the "damage". This in turnhas an adverse effect on the sharpness of the other item parameterestimates. Removal of such items thus improves classificationperformance.

Before removing an item from the test item pool, the item should bechecked as to whether it is properly specified as to type. For example,suppose an item is distributed as Bernoulli within both its partitions.Further, suppose that the item type is specified as AB in the posetmodel of FIG. 8 and the probability of a positive response for testsubjects in the PDOI of state AB is significantly less than 1. It may bethat the proper specification is actually a state higher than AB in thedomain model such as ABC, ABD, or the union of ABC and ABD. On the otherhand, suppose that the probability of a positive response for testsubjects in a state lower than AB in the model is suspiciously high. Theproper specification may actually be a state lower than AB such as A orB. The proper procedure to follow in such situations is to vary the typespecification of the test item with the objective of finding aspecification for which test subjects in the PDOI of the typedesignation provide a positive response with a probability reasonablyclose to 1 and test subjects in states less than the type designationprovide a positive response reasonably close to 0.

Another possibility to be considered before removing an item from thetest item pool is whether the response distribution itself may bemisspecified. If certain erroneous responses occur frequently, it may beappropriate to consider more complex response distributions. Forexample, a Bernoulli distribution might be extended to a multinomialdistribution. Responses from the training sample of test subjectsprovide information as to frequencies of occurrence of the differentresponses. Histograms of these response frequencies can be constructedbased on the classification results. Distributions can then bereformulated and the new parameters estimated.

Analysis of erroneous responses can speed classification. Supposeinitially that an item has a two-element partition (one a PDOI) and thatthe response distributions are Bernoulli. Further, suppose that acertain erroneous response gives strong evidence that a test subject hasa particular lack of functionality and/or lack of possession of aparticular set of facts. The response distribution for test subjects inthe complement of the PDOI may be respecified to a multinomial to takeinto account this particular erroneous response. The responsedistribution for states not in the complement does not necessarily haveto be changed as well. A third partition within the complement set maybe created to reflect that the particular erroneous response stronglysuggests membership in it. Hence, rather than grouping all erroneousresponses as negative, information about such responses can be exploitedin the classification process. The above scenario illustrates theutility in being able to respecify item type and/or the type of responsedistributions.

If a test item is under suspicion initially, then a well-designedtraining sample should replicate the item's type in question to directlycompare performance of the estimates. Examining similarly constructeditems should help in verifying both the type specification and assessingrelative performance.

Model fitting and item analysis are very much intertwined. An importantconsideration in evaluating item performance is the possibility that themodel itself may be misspecified. A misspecification in the model maylead to parameter estimates for an item that are not sharp when theyactually should be. This can happen if a state representing an importantcognitive ability or function is hidden and a misspecified item involvesthat ability or function. The misspecified item may in fact be very goodin terms of differentiating knowledge or functionality levels. Bycorrecting the model, this may become apparent. On the other hand,models may not appear to fit well due to poor items. Thus, bothpossibilities must be examined simultaneously.

A way of determining whether a model is a good fit is to analyze theclassification performance assuming that the items are all gooddiscriminators having sharp distributional properties and that they arespecified correctly. Consider the description below as an illustration.For the sake of discussion, suppose that the items are associated with astate and have two partitions.

For example, let the true model be the poset shown in FIG. 9 and thespecified model be the poset shown in FIG. 8. In this situation state ABis superfluous. Suppose the items specified as type AB (if they exist)are really of type {ABC,ABD}, the union of ABC and ABD. Test subjectsshould then be correctly classified and few will be classified to stateAB.

For a second example, suppose that the poset in FIG. 8 is the true modeland the one in FIG. 9 is the specified model. The state AB is hiddeninsofar as the specified model is concerned. Let the items truly of typeAB be specified as {ABC,ABD}. Eventually, the mass will be distributedamong the states that act as "covers" of AB and those states that are"covered" by AB, i.e. ABC, ABD, A, and B, provided the appropriate itemtypes are administered. If the item pool does not contain an item trulyof type AB, test subjects not in AB will be classified without problems.If the test subjects are in AB, again mass will eventually bedistributed to the states directly above and below AB, depending on theitem selection sequence.

For a third example, when a hidden state from the poset shown in FIG. 8is a cover of 0 such as A and there are no A-type items, test subjectsin state A will most likely be classified to 0. If items truly of type Aare present in the test item pool and specified as {AB,AC,AD}, the samephenomenon as in the second example may occur.

For a fourth example, let the poset shown in FIG. 10 be the specifiedmodel, again assuming the true model is the poset shown in FIG. 8. Witha large missing section as illustrated in FIG. 10, it is more difficultto discover the true model. The conclusions about mass distribution withstates missing from the poset can be generalized to apply to specifiedmodels having large missing sections. It is necessary to identify whatthe covers would be for each missing state among the states within thespecified model and the states within the specified model that would becovered by each missing state if it were included in the model. Buildingup large missing sections may require several steps in model fitting.

For a fifth example, assume that a test subject is meandering, i.e. hisor her response pattern is erratic involving a series of responses thatare contradictory. This situation cannot be explained either by impropertype-specification of the test items or by an ill-fitting model.

It follows from the above examples that if items are sharp and correctlyspecified for the model at hand and if the model is overspecified withtoo many states, those that are not really valid in a practical senseshould have few test subjects classified to them and should be removedfrom the model. On the other hand, if a state is missing, thenclassification should in most cases not be strong to any state for testsubjects belonging to that missing state.

Erratic response patterns should be carefully examined. A significantnumber of test subjects sharing similar aberrant patterns (i.e. patternsthat do not lead to a dominant posterior value emerging) strengthens theevidence that a state may be missing and should indicate its location inthe poset. Test subjects that meander with inconsistent responsepatterns may need to be eliminated from the trial sample.

The description of the invention has thus far focused on domain modelsbased on discrete posets as exemplified in FIGS. 1, 5, 8, 9, and 10.Domain models can also be based on a combination of a discrete poset anda subset of Euclidean space. In the combination model, test subjectstates are represented by both a state s in the poset and the value of acontinuous parameter t which may be multi-dimensional. For example, theparameter t might correspond to the intelligence quotient of the testsubject. A sensible partial order for elements in the combination modelwould be to let (s₁,t₁) be lower than or equal to (s₂,t₂) if and only ifs₁ ≦s₂ in the discrete poset and t₁ ≦t₂ in the subset of Euclideanspace. Items not involving the cognitive or functionality attributesrepresented by t still partition the discrete component as before. Itemsinvolving the attributes represented by t can partition the discreteposet component according to whether the discrete state classconditional densities given a value of t coincide for all values of t.Thus, the mapping to determine separation of the states in the discreteposet component given a pool of test items can be applied in the contextof the combination model as well.

The responses for a finite poset model might be distributed discretelyor come from a continuous distribution such as a normal distribution.Moreover, the distributions within the same item's partitions may vary.For example, one partition may be multinomially distributed, while theother may be a Bernoulli trial. For combination models, the responsesgiven a state in those models may also be discrete or continuous. Again,distribution types between an item's partitions may be different for thecombination model. An example of a continuous response distribution is anormal distribution with mean t and variance σ².

The class conditional densities f_(i) and g_(i) for a combinationposet-Euclidean space model might be

    f.sub.i (X=1|s,t)=H(t-α.sub.1) if e(i)≦s

    g.sub.i (X=1|s,t)=H(t-α.sub.2) if e(i)≦/s(25)

where X=1|s,t denotes a positive response given s and t and ##EQU20##

The parameter p is a known quantity in the range from 0 to 1 and t isreal-valued. Note how the distributions associated with the densitiesf_(i) and g_(i) reflect the underlying order structure of thecombination model.

The same statistical framework applies as before. The initial SPS (priordistribution) is a probability density function and is given by

    π.sub.0 (s,t)=π.sub.0 (t|s)π.sub.0 (s)   (27)

where the first term of the product is the conditional probabilitydensity value of t given s and the second term is the marginalprobability of s.

Posterior distributions π_(n) (s,t|x_(n)) can be calculated by Bayesrule given response path x_(n). An example of a loss function for acombination model is ##EQU21## where the state d resulting from theapplication of a particular decision rule is a function of the finalπ_(n)(s,t|x_(n)) and implies the values s and t. The constant K is ascaling parameter greater than 0.

Strategy trees are applicable to combination models as well. Quantitieswhich can provide a computationally simple basis for stopping in the"peel-back" context described above are the posterior variance(s) of t:##EQU22## where the mean of the conditional distribution is given by

    u.sub.n (s)=∫tπn.sub.n (t|s)dt            (30)

and the marginal posterior values for s in S are given by

    π.sub.n (s)=∫π.sub.n (s,t)dt=∫π.sub.n (t|s)π.sub.n (s)dt                            (31)

Note that the probabilities of test subjects being in partitions of thediscrete poset component can be calculated using the marginal posteriorvalues. These probabilities can be used in item selection rules such asin equation (4). Also, the mapping Φ can still be applied on states in Sas well as for elements in the combination model, for instance, byemploying weighted discrepancy measures on the class conditionaldensities given values of t.

The equation for sh₁ (π_(n),i) becomes

    sh.sub.1 (π.sub.n,i)=∫En(π.sub.(n+1) |X.sub.(n+1) =x,It.sub.(n+1) =i)P(X.sub.(n+1) =x|π.sub.n,It.sub.(n+1) =i)dx(32)

where ##EQU23##

The equation for Kg₁ (π_(n),i) becomes ##EQU24## where f_(i) (x|s,t) isthe response distribution density value at x of item i given s and t.The quantity π.sub.(n+1) (s,t|X.sub.(n+1) =x, It.sub.(n+1) =i) isproportional to f_(i) (x|s,t)π_(n) (s,t).

The k-step version of the above equation is ##EQU25## where "min over j"means the value of the quantity in brackets for an item j from thepreceding available item pool which minimizes the value.

In general, all of the item selection, stopping, decision, and updatingrules generalize straightforwardly for use with the combination model.Also, fixed tests can be designed, and ideal patterns can be employed.

An important application of the system and of the combination model inparticular is in medical diagnosis. The application of the aboveclassification framework and techniques is completely analogous. Itemscan be viewed as experiments or tests, and the underlying model canrepresent various conditions related by a partial order structure. Forinstance, three states can be used to represent a diabetes condition,with one state representing no condition, another state representingType I diabetes (insulin-dependent), and a third state representing TypeII diabetes (non-insulin-dependent). The ordering between these statesis natural in relation to the no-condition state, and for the sake ofdiscussion, the states representing Type I and Type II respectively willbe assumed to be incomparable. Further, the corresponding classconditional response distributions of experiments such as measuringglucose levels in blood should reflect that ordering, in the sense thatthe higher the observed responses, the more likely a diabetes conditionexists. It is when the response distributions reflect the underlyingorder structure of the model that the system works most effectively, sothat medical applications are well-suited for the system. Of course, itis likely that the class conditional response distributions for the TypeI and Type II conditions differ. Other experiments which separate statesmay be necessary to increase the accuracy in diagnosis. As an example ofa multi-dimensional continuous response, an experiment may measure thelevels of glucose and ketones together, where ketones are smallfragments from the breakdown of fatty acids. The corresponding classconditional densities would then be multivariate. If the levels ofglucose and ketones are interrelated, then it may indeed be moreinformative to measure them together.

Perhaps the diabetes condition or other medical conditions could moreappropriately be modeled as a continuous variable within a discretestate such as in a combination model. In general, the poset orcombination model can get more complex as various auxiliary conditionsare introduced which may or may not be present in conjunction with otherconditions, and each of which may have their own natural ordering. Notehow the auxiliary conditions may themselves have an effect on experimentresponses. This can be modelled by specifying an appropriate partitionfor an experiment in terms of class conditional densities.

Various types of experiments having a variety of class conditionalresponse distribution types can be administered sequentially which mayhave different objectives in terms of what they measure. Costs ofobservation for the experiments can be incorporated into the analysis,as before. Hence, not only can this system be used to conduct actualmedical diagnosis, but it can be used to gauge the cost-effectiveness ofthe experiments. Experiments whose contribution to the classificationprocess do not on average overcome their cost of observation can beidentified as they will not be selected with a high weighted frequencyand/or probability of administration for instance with respect to SPSsand class conditional densities.

What is claimed is:
 1. A method for classifying a test subject in one ofa plurality of states in a domain, a domain being a set of facts, aquality measure having a range of values, or a combination of a set offacts and a quality measure, the set of facts for a knowledge domainbeing any set of facts, the set of facts for a functionality domainbeing a set of facts relating to the functionality of a test subject, astate being characterized by a subset of facts, a value in the range ofvalues for a quality measure, or a combination of a subset of facts anda value for a quality measure, a first state being higher than or equalto a second state and a second state being lower than or equal to afirst state if (1) the subset of facts or the quality measure valueassociated with the first state respectively includes the subset offacts or is greater than or equal to the quality measure valueassociated with the second state or (2) the subset of facts and thequality measure value associated with the first state respectivelyincludes the subset of facts and is greater than or equal to the qualitymeasure value associated with the second state, a test subject beingclassified in the highest state of which he has the knowledge orfunctionality, the method comprising the steps:specifying a domaincomprising a plurality of states and determining thehigher-lower-neither relationships for each state, thehigher-lower-neither relationships for a state being a specification ofwhich states are higher, which states are lower, and which states areneither higher or lower, the plurality of states including a first,second, and third fact state characterized by subsets of facts wherein(1) the first and second fact states are higher than the third factstate and the first fact state is neither higher nor lower than thesecond fact state or (2) the first fact state is higher than the secondand third fact states and the second fact state is neither higher norlower than the third fact state; specifying a test item pool comprisinga plurality of test items, a test item engendering a response whenadministered to a test subject; specifying an initial state probabilityset (SPS) for the test subject to be classified, each member of theinitial SPS being an initial estimate of the probability density valuethat the test subject is associated with a particular state in thedomain; specifying a class conditional density f_(i) (x|s) for each testitem i in the test item pool for each state s in the domain, a classconditional density being a specification of the probability of a testsubject in state s providing a response x to the test item i, each itempartitioning the domain of states into a plurality of partitionsaccording to the class conditional densities associated with the item, apartition being a subset of states for which the class conditionaldensities are the same or the union of such subsets; determining theclassification of the test subject.
 2. The method of claim 1 wherein thestep of determining the classification of the test subject includes thesteps:administering one of a sequence of test items from the test itempool to the test subject; updating the SPS after receiving a response tothe administered test item.
 3. The method of claim 2 wherein the(n+1)'th administered test item i.sub.(n+1) in the step of administeringa sequence of test items is selected from test items in the test itempool that have not already been administered, the test item i.sub.(n+1)being the test item which results in a partition for which theprobability of a test subject being in the partition is nearest to 0.5.4. The method of claim 2 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of administering a sequence of test items isselected from test items in the test item pool that have not alreadybeen administered, i.sub.(n+1) being the test item that results in thelargest value of a weighted uncertainty measure, the uncertainty measurebeing a measure of the uncertainty as to which of the test item'spartitions that the test subject is in, the uncertainty measure beingsmallest when all but one of the partition probabilities are near 0, apartition probability being the probability of the test subject being inthe partition.
 5. The method of claim 2 wherein the (n+1)'thadministered test item i.sub.(n+1) in the step of administering asequence of test items is selected from test items in the test item poolthat have not already been administered, i.sub.(n+1) being the test itemthat results in the smallest value of a weighted SPS uncertainty measuregiven the hypothetical administration of i.sub.(n+1), an SPS uncertaintymeasure being a minimum when all but one of the SPS probability densityvalues are near
 0. 6. The method of claim 2 wherein the (n+1)'thadministered test item i.sub.(n+1) in the step of administering asequence of test items is selected from test items in the test item poolthat have not already been administered, i.sub.(n+1) being the test itemthat results in the smallest value of a weighted SPS uncertainty measuregiven the hypothetical administration of the sequence of test itemsi.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k), k being an integer, anSPS uncertainty measure being a minimum when all but one of the SPSprobability density values are near
 0. 7. The method of claim 2 whereinthe (n+1)'th administered test item i.sub.(n+1) in the step ofadministering a sequence of test items is selected from test items inthe test item pool that have not already been administered, i.sub.(n+1)being the test item that results in the largest value of a weighteddistance measure between the SPS after a hypothetical administration ofan (n+1)'th test item and the SPS after the actual administration of then'th test item, the distance measure being a measure of the differencesin the two SPSs.
 8. The method of claim 2 wherein the (n+1)'thadministered test item i.sub.(n+1) in the step of administering asequence of test items is selected from test items in the test item poolthat have not already been administered, i.sub.(n+1) being the test itemthat results in the largest value of a weighted distance measure betweenthe SPS after a hypothetical administration of the sequence of testitems i.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k), k being an integer,and the SPS after the actual administration of the n'th test item, thedistance measure being a measure of the differences in the two SPSs. 9.The method of claim 2 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of administering a sequence of test items isselected from test items in the test item pool that have not alreadybeen administered, the test item i.sub.(n+1) being the test itemcorresponding to the largest value of a weighted discrepancy measuresummed over all pairs of states, a discrepancy measure for a test itemgiven two states being a measure of the distance between the classconditional densities for the item and the two states.
 10. The method ofclaim 9 wherein the (n+1)'th administered test item i.sub.(n+1) in thestep of administering a sequence of test items is selected from testitems in the test item pool that have not already been administered, thetest item i.sub.(n+1) being selected using a two-valued function Φ, thefunction Φ being a function of (1) a test item and (2) a first state anda second state, Φ having a first value if the test item separates thefirst and second states, Φ having a second value if the test item doesnot separate the first and second states.
 11. The method of claim 10wherein Φ has a first value for a plurality of the test items for aspecified first state and a specified second state, the test itemi.sub.(n+1) being selected randomly from the plurality of test items.12. The method of claim 2 wherein test item i'.sub.(n+1) is tentativelyselected as the (n+1)'th administered test item i.sub.(n+1) in the stepof administering a sequence of test items, test item i'.sub.(n+1) beingselected using a predetermined selection rule from test items in thetest item pool that have not already been administered, a randomdecision being made either to use test item i'.sub.(n+1) as the (n+1)'thadministered test item i.sub.(n+1) or to select another test item fromthe test item pool.
 13. The method of claim 12 wherein the test itemsare ordered according to a goodness criterion associated with thepredetermined selection rule, the test item i'.sub.(n+1) being the besttest item, a plurality of the next-in-order test items being denoted asthe better test items, one of the better test items being selected asthe (n+1)'th administered test item i.sub.(n+1) if the decision is madeto select a test item other than test item i'.sub.(n+1).
 14. The methodof claim 13 wherein the selection of one of the better test items israndomly made, the random selection being biased in accordance with theorder of the better test items.
 15. The method of claim 13 wherein thedecision to select a test item other than test item i'.sub.(n+1) is madewith a probability that is a function of a "temperature" parameter, theprobability being a monotonic function of "temperature".
 16. The methodof claim 2 wherein the (n+1)'th administered test item i.sub.(n+1) inthe step of administering a sequence of test items is selected from testitems in the test item pool that have not already been administered, thetest item i(r) being the test item that would be selected usingselection rule r, the index r denoting any one of a plurality ofselection rules, the test item i.sub.(n+1) being the test item i(r) thatresults in a condition selected from the group of conditions (1)smallest value for a weighted loss function, (2) smallest value for aweighted uncertainty measure, and (3) largest value for a weighteddistance measure.
 17. The method of claim 2 wherein the (n+1)'thadministered test item i.sub.(n+1) in the step of administering asequence of test items is selected from test items in the test item poolthat have not already been administered, the test item i(r) being thetest item that would be selected using selection rule r, the index rdenoting any one of a plurality of selection rules, the test itemi.sub.(n+1) being a random selection from the test items i(r).
 18. Themethod of claim 2 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of administering a sequence of test items isselected from test items in the test item pool that have not alreadybeen administered, the test item i.sub.(n+1) being the test item thatmaximizes a weighted relative ranking measure based on a plurality ofitem selection rules, a weighted relative ranking measure being aweighted function of the relative rankings of attractiveness for eachtest item with respect to a plurality of item selection rules.
 19. Themethod of claim 2 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of administering a sequence of test items isselected from test items in the test item pool that have not alreadybeen administered, the test item i.sub.(n+1) being the test itemcorresponding to the largest value of the sum of π_(n) (j)π_(n)(k)d_(jk) (i) over all states j and k in the domain, π_(n) (j) denotingthe members of the updated SPS after the test subject has responded tothe n'th administered test item, d_(jk) (i) denoting a measure of thedegree of discrimination between states j and k provided by test item ias measured by a discrepancy measure on the corresponding classconditional densities.
 20. The method of claim 19 wherein d_(jk) (i) isa first function of an integral of a second function over all possibleresponses to test item i, the second function being the absolute valueof the difference in a first class conditional density and a secondclass conditional density, the first class conditional density beingassociated with item i and state j and the second class conditionaldensity being associated with item i and state k.
 21. The method ofclaim 20 wherein the first function equals zero when the first andsecond class conditional densities are the same, the first functionbeing equal to a positive constant when the first and second classconditional densities are different.
 22. The method of claim 19 whereind_(jk) (i) is a first function of an integral of a second function overall possible responses to test item i, the second function being a thirdfunction of a difference function, the difference function being thedifference between a fourth function of the first class conditionaldensity and the fourth function of the second class conditional density,the first class conditional density being associated with item i andstate j and the second class conditional density being associated withitem i and state k, the variety of functions denoted by the term"function" including the identity function wherein the function of aquantity is equal to the quantity.
 23. The method of claim 19 whereind_(jk) (i) is a first function of an integral of a second function overall possible responses to test item i, the second function being afunction of the ratio of a third function of a first class conditionaldensity and the third function of a second class conditional densityweighted by the first class conditional density when the first andsecond class conditional densities are different, the first classconditional density being associated with item i and state j and thesecond class conditional density being associated with item i and statek, d_(jk) (i) being equal to zero when the first and second classconditional densities are the same, the variety of functions denoted bythe term "function" including the identity function wherein the functionof a quantity is equal to the quantity.
 24. The method of claim 2wherein the (n+1)'th administered test item i.sub.(n+1) in the step ofadministering a sequence of test items is selected from test items inthe test item pool that have not already been administered, i.sub.(n+1)being the test item that results in the smallest value of a weightedloss function for k=1, a loss function being a function of (1) the statein the domain, (2) a classification decision action that specifies astate, and (3) the number k of test items to be administered beginningwith i.sub.(n+1).
 25. The method of claim 2 wherein the (n+1)'thadministered test item i.sub.(n+1) in the step of administering asequence of test items is selected from test items in the test item poolthat have not already been administered, i.sub.(n+1) being the test itemthat results in the smallest value of a weighted loss function given thehypothetical administration of the sequence of test items i.sub.(n+1),i.sub.(n+2), . . . , i.sub.(n+k), k being an integer, a loss functionbeing a function of (1) the state in the domain, (2) a classificationdecision action that specifies a state, and (3) the number k of testitems to be administered beginning with i.sub.(n+1).
 26. The method ofclaim 2 wherein the (n+1)'th administered test item i.sub.(n+1) in thestep of administering a sequence of test items is selected from testitems in the test item pool that have not already been administered,i.sub.(n+1) being the test item for which a weighted loss inadministering the next k test items is the smallest, the loss inadministering k test items being defined by a loss function consistingof two additive components, the first component being a measure of theloss associated with the classification of the test subject afteradministering the k test items, the loss associated with an incorrectclassification being higher than the loss associated with a correctclassification, the second component being the cost of administering thek test items.
 27. The method of claim 26 wherein the first component ofthe loss function is (1) a constant A₁ (s) if the test subject would beclassified correctly after administering k additional test items and (2)a constant A₂ (s) if the test subject would be classified incorrectlyafter administering k additional test items, the constants A₁ (s) and A₂(s) having a possible dependence on the state s, the second component ofthe loss function being the sum of the individual costs of administeringthe k additional test items.
 28. The method of claim 2 wherein therespective responses to test items i.sub.(n+1), i.sub.(n+2), . . . ,i.sub.(n+k) are x.sub.(n+1), x.sub.(n+2), . . . , x.sub.(n+k), k beingan integer, and the SPS updating rule is a function of the classconditional densities evaluated at x.sub.(n+1), x.sub.(n+2), . . . ,x.sub.(n+k) and a given SPS with the SPS probability density value for astate being nondecreasing in the class conditional density value forfixed SPS and fixed class conditional density values for all otherstates.
 29. The method of claim 2 wherein the step of determining theclassification of the test subject includes the steps:applying astopping rule, the steps of claim 2 being repeated if the stopping ruledoes not require that the administering of test items be terminated;classifying the test subject as to a state in the domain in accordancewith a decision rule if the stopping rule requires that theadministering of test items be terminated.
 30. The method of claim 29wherein the domain is a combination model and the stopping rule is tostop administering test items after administering the n'th test itemi_(n) if the marginal posterior value for a state in the discretecomponent of the domain is greater than a first predetermined value andthe posterior variance of the continuous parameter in the domain is lessthan a second predetermined value.
 31. The method of claim 29 whereinthe stopping rule is to stop administering test items if a weighteduncertainty measure with respect to the SPS after administration of then'th test item i_(n) is less than a predetermined value.
 32. The methodof claim 29 wherein the stopping rule is to stop administering testitems if a weighted distance measure between the initial SPS and the SPSafter administration of the n'th test item i_(n) exceeds a predeterminedvalue.
 33. The method of claim 29 wherein the stopping rule is to stopadministering test items after administering the n'th test item i_(n) ifa weighted loss function is less than a predetermined value.
 34. Themethod of claim 29 wherein the stopping rule is to stop administeringtest items if the largest value of the SPS after administration of then'th test item i_(n) exceeds a threshold.
 35. The method of claim 29wherein the stopping rule is to stop administering test items if apredetermined number of test items have been administered.
 36. Themethod of claim 29 wherein the stopping rule is to stop administeringtest items after administering the n'th test item i_(n) if, given thehypothetical administration of the sequence of one or more test itemsi.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k) where k is an integer, aweighted loss function increases in value.
 37. The method of claim 29wherein the stopping rule is to stop administering test items afteradministering the n'th test item i_(n) if, given the hypotheticaladministration of the sequence of one or more test items i.sub.(n+1),i.sub.(n+2), . . . , i.sub.(n+k) where k is an integer, a weighteduncertainty measure decreases by less than a predetermined value, theweighted uncertainty measure being with respect to the SPS after theadministration of item i_(n) and after the hypothetical administrationof the sequence of test items i.sub.(n+1), i.sub.(n+2), . . . ,i.sub.(n+k).
 38. The method of claim 29 wherein the stopping rule is tostop administering test items after administering the n'th test itemi_(n) if, given the hypothetical administration of the sequence of oneor more test items i.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k) where kis an integer, a weighted distance measure increases by less than apredetermined value, the weighted distance measure being with respect tothe SPS after the administration of item i_(n) and after thehypothetical administration of the sequence of test items i.sub.(n+1),i.sub.(n+2), . . . , i.sub.(n+k).
 39. The method of claim 29 wherein thestopping rule is to stop administering test items after administeringthe n'th test item i_(n) if a specified condition occurs, the specifiedcondition being defined in terms of one or more first conditions, one ormore second conditions, one or more third conditions, one or more fourthconditions, one or more fifth conditions, one or more sixth conditions,or combinations thereof, a first condition being that a weighted lossfunction is less than a first predetermined value, a second conditionbeing that a weighted loss function increases in value given thehypothetical administration of one or more test items, a third conditionbeing that a weighted uncertainty measure is less than a secondpredetermined value, a fourth condition being that a weighteduncertainty measure decreases by less than a third predetermined valuegiven the hypothetical administration of one or more test items, a fifthcondition being that a weighted distance measure is larger than a fourthpredetermined value, and a sixth condition being that a weighteddistance measure increases by less than a predetermined value given thehypothetical administration of one or more test items.
 40. The method ofclaim 29 wherein the decision rule is to select the state associatedwith the highest value in the SPS.
 41. The method of claim 29 whereinthe decision rule is to select the state associated with the smallestvalue for a weighted loss function.
 42. The method of claim 1 furthercomprising the step:specifying a strategy tree to be used inadministering test items to test subjects, a strategy tree being aplurality of paths, each path beginning with the first test item to beadministered, continuing through a sequence alternating between aparticular response to the last test item and the specification of thenext test item, and ending with a particular response to the final testitem in the path, the classification of the test subject being specifiedfor each path of the strategy tree using a decision rule.
 43. The methodof claim 42 further comprising the step:determining for a test item inthe test item pool the weighted frequency and/or the probability ofbeing administered.
 44. The method of claim 43 further comprising thestep:removing a test item from the test item pool if the weightedfrequency and/or the probability of being administered is less than apredetermined value.
 45. The method of claim 42 wherein the (n+1)'thadministered test item i.sub.(n+1) in the step of specifying a strategytree is selected from test items in the test item pool that have notalready been administered, the test item i.sub.(n+1) being the test itemwhich results in a partition for which the probability of a test subjectbeing in the partition is nearest to 0.5.
 46. The method of claim 42wherein the (n+1)'th administered test item i.sub.(n+1) in the step ofspecifying a strategy tree is selected from test items in the test itempool that have not already been administered, i.sub.(n+1) being the testitem that results in the largest value of a weighted uncertaintymeasure, the uncertainty measure being a measure of the uncertainty asto which of the test item's partitions that the test subject is in, theuncertainty measure being smallest when all but one of the partitionprobabilities are near 0, a partition probability being the probabilityof the test subject being in the partition.
 47. The method of claim 42wherein the (n+1)'th administered test item i.sub.(n+1) in the step ofspecifying a strategy tree is selected from test items in the test itempool that have not already been administered, i.sub.(n+1) being the testitem that results in the smallest value of a weighted SPS uncertaintymeasure, given the hypothetical administration of i.sub.(n+1), an SPSuncertainty measure being a minimum when all but one of the SPSprobability density values are near
 0. 48. The method of claim 42wherein the (n+1)'th administered test item i.sub.(n+1) in the step ofspecifying a strategy tree is selected from test items in the test itempool that have not already been administered, i.sub.(n+1) being the testitem that results in the smallest value of a weighted SPS uncertaintymeasure given the hypothetical administration of the sequence of testitems i.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k), k being an integer,an SPS uncertainty measure being a minimum when all but one of the SPSprobability density values are near
 0. 49. The method of claim 42wherein the (n+1)'th administered test item i.sub.(n+1) in the step ofspecifying a strategy tree is selected from test items in the test itempool that have not already been administered, i.sub.(n+1) being the testitem that results in the largest value of a weighted distance measurebetween the SPS after a hypothetical administration of an (n+1)'th testitem and the SPS after the actual administration of the n'th test item,the distance measure being a measure of the differences in the two SPSs.50. The method of claim 42 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of specifying a strategy tree is selected fromtest items in the test item pool that have not already beenadministered, i.sub.(n+1) being the test item that results in thelargest value of a weighted distance measure between the SPS after ahypothetical administration of the sequence of test items i.sub.(n+1),i.sub.(n+2), . . . , i.sub.(n+k), k being an integer, and the SPS afterthe actual administration of the n'th test item, the distance measurebeing a measure of the differences in the two SPSs.
 51. The method ofclaim 42 wherein the (n+1)'th administered test item i.sub.(n+1) in thestep of specifying a strategy tree is selected from test items in thetest item pool that have not already been administered, the test itemi.sub.(n+1) being the test item corresponding to the largest value of aweighted discrepancy measure summed over all pairs of states, adiscrepancy measure for a test item given two states being a measure ofthe distance between the class conditional densities for the item andthe two states.
 52. The method of claim 51 wherein the (n+1)'thadministered test item i.sub.(n+1) in the step of specifying a strategytree is selected from test items in the test item pool that have notalready been administered, the test item i.sub.(n+1) being selectedusing a two-valued function Φ, the function Φ being a function of (1) atest item and (2) a first state and a second state, Φ having a firstvalue if the test item separates the first and second states, Φ having asecond value if the test item does not separate the first and secondstates.
 53. The method of claim 52 wherein Φ has a first value for aplurality of the test items for a specified first state and a specifiedsecond state, the test item i.sub.(n+1) being selected randomly from theplurality of test items.
 54. The method of claim 42 wherein test itemi'.sub.(n+1) is tentatively selected as the (n+1)'th administered testitem i.sub.(n+1) in the specifying a strategy tree, test itemi'.sub.(n+1) being selected using a predetermined selection rule fromtest items in the test item pool that have not already beenadministered, a random decision being made either to use test itemi'.sub.(n+1) as the (n+1)'th administered test item i.sub.(n+1) or toselect another test item from the test item pool.
 55. The method ofclaim 54 wherein the test items are ordered according to a goodnesscriterion associated with the predetermined selection rule, the testitem i'.sub.(n+1) being the best test item, a plurality of thenext-in-order test items being denoted as the better test items, one ofthe better test items being selected as the (n+1)'th administered testitem i.sub.(n+1) if the decision is made to select a test item otherthan test item i'.sub.(n+1).
 56. The method of claim 55 wherein theselection of one of the better test items is randomly made, the randomselection being biased in accordance with the order of the better testitems.
 57. The method of claim 55 wherein the decision to select a testitem other than test item i'.sub.(n+1) is made with a probability thatis a function of a "temperature" parameter, the probability being amonotonic function of "temperature".
 58. The method of claim 42 whereinthe (n+1)'th administered test item i.sub.(n+1) in the step ofspecifying a strategy tree is selected from test items in the test itempool that have not already been administered, the test item i(r) beingthe test item that would be selected using selection rule r, the index rdenoting any one of a plurality of selection rules, the test itemi.sub.(n+1) being the test item i(r) that results in a conditionselected from the group of conditions (1) smallest value for a weightedloss function, (2) smallest value for a weighted uncertainty measure,and (3) largest value for a weighted distance measure.
 59. The method ofclaim 42 wherein the (n+1)'th administered test item i.sub.(n+1) in thestep of specifying a strategy tree is selected from test items in thetest item pool that have not already been administered, the test itemi(r) being the test item that would be selected using selection rule r,the index r denoting any one of a plurality of selection rules, the testitem i.sub.(n+1) being a random selection from the test items i(r). 60.The method of claim 42 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of specifying a strategy tree is selected fromtest items in the test item pool that have not already beenadministered, the test item i.sub.(n+1) being the test item thatmaximizes a weighted relative ranking measure based on a plurality ofitem selection rules, a weighted relative ranking measure being aweighted function of the relative rankings of attractiveness for eachtest item with respect to a plurality of item selection rules.
 61. Themethod of claim 42 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of specifying a strategy tree is selected fromtest items in the test item pool that have not already beenadministered, the test item i.sub.(n+1) being the test itemcorresponding to the largest value of the sum of π_(n) (j)π_(n)(k)d_(jk) (i) over all states j and k in the domain, π_(n) (j) denotingthe members of the updated SPS after the test subject has responded tothe n'th administered test item, d_(jk) (i) denoting a measure of thedegree of discrimination between states j and k provided by test item ias measured by a discrepancy measure on the corresponding classconditional densities.
 62. The method of claim 61 wherein d_(jk) (i) isa first function of an integral of a second function over all possibleresponses to test item i, the second function being the absolute valueof the difference in a first class conditional density and a secondclass conditional density, the first class conditional density beingassociated with item i and state j and the second class conditionaldensity being associated with item i and state k.
 63. The method ofclaim 62 wherein the first function equals zero when the first andsecond class conditional densities are the same, the first functionbeing equal to a positive constant when the first and second classconditional densities are different.
 64. The method of claim 61 whereind_(jk) (i) is a first function of an integral of a second function overall possible responses to test item i, the second function being a thirdfunction of a difference function, the difference function being thedifference between a fourth function of the first class conditionaldensity and the fourth function of the second class conditional density,the first class conditional density being associated with item i andstate j and the second class conditional density being associated withitem i and state k, the variety of functions denoted by the term"function" including the identity function wherein the function of aquantity is equal to the quantity.
 65. The method of claim 61 whereind_(jk) (i) is a first function of an integral of a second function overall possible responses to test item i, the second function being afunction of the ratio of a third function of a first class conditionaldensity and the third function of a second class conditional densityweighted by the first class conditional density when the first andsecond class conditional densities are different, the first classconditional density being associated with item i and state j and thesecond class conditional density being associated with item i and statek, d_(jk) (i) being equal to zero when the first and second classconditional densities are the same, the variety of functions denoted bythe term "function" including the identity function wherein the functionof a quantity is equal to the quantity.
 66. The method of claim 42wherein the (n+1)'th administered test item i.sub.(n+1) in the step ofspecifying a strategy tree is selected from test items in the test itempool that have not already been administered, i.sub.(n+1) being the testitem that results in the smallest value of a weighted loss functiongiven the hypothetical administration of the sequence of test itemsi.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k), k being an integer, aloss function being a function of (1) the state in the domain, (2) aclassification decision action that specifies a state, and (3) thenumber k of test items to be administered beginning with i.sub.(n+1).67. The method of claim 42 wherein the (n+1)'th administered test itemi.sub.(n+1) in the step of specifying a strategy tree is selected fromtest items in the test item pool that have not already beenadministered, i.sub.(n+1) being the test item that results in thesmallest value of a weighted loss function for k=1, a loss functionbeing a function of (1) the state in the domain, (2) a classificationdecision action that specifies a state, and (3) the number k of testitems to be administered beginning with i.sub.(n+1).
 68. The method ofclaim 42 wherein the (n+1)'th administered test item i.sub.(n+1) in thestep of specifying a strategy tree is selected from test items in thetest item pool that have not already been administered, i.sub.(n+1)being the test item for which the weighted loss in administering thenext k test items is the smallest, the loss in administering k testitems being defined by a loss function consisting of two additivecomponents, the first component being a measure of the loss associatedwith the classification of the test subject after administering the ktest items, the loss associated with an incorrect classification beinghigher than the loss associated with a correct classification, thesecond component being the cost of administering the k test items. 69.The method of claim 68 wherein the first component of the loss functionis (1) a constant A₁ (s) if the test subject would be classifiedcorrectly after administering k additional test items and (2) a constantA₂ (s) if the test subject would be classified incorrectly afteradministering k additional test items, the constants A₁ (s) and A₂ (s)having a possible dependence on the state s, the second component of theloss function being the sum of the individual costs of administering thek additional test items.
 70. The method of claim 42 wherein in the stepof specifying a strategy tree a stopping rule is applied after eachspecification of a test item in the strategy tree, an additional testitem being specified only if the stopping rule so specifies.
 71. Themethod of claim 70 wherein the domain is a combination model and thestopping rule is to stop administering test items after administeringthe n'th test item i_(n) if the marginal posterior value for a state inthe discrete component of the domain is greater than a firstpredetermined value and the posterior variance of the continuousparameter in the domain is less than a second predetermined value. 72.The method of claim 70 wherein the stopping rule is to stop specifyingtest items if the largest value of the SPS after administration of then'th test item i_(n) exceeds a threshold.
 73. The method of claim 70wherein the stopping rule is to stop specifying test items if apredetermined number of test items have been specified.
 74. The methodof claim 70 wherein the stopping rule is to stop administering testitems if a weighted uncertainty measure with respect to the SPS afteradministration of the n'th test item i_(n) is less than a predeterminedvalue.
 75. The method of claim 70 wherein the stopping rule is to stopadministering test items if a weighted distance measure between theinitial SPS and the SPS after administration of the n'th test item i_(n)exceeds a predetermined value.
 76. The method of claim 70 wherein thestopping rule is to stop administering test items after administeringthe n'th test item i_(n) if a weighted loss function is less than apredetermined value.
 77. The method of claim 70 wherein the stoppingrule is to stop administering test items after administering the n'thtest item i_(n) if, given the hypothetical administration of thesequence of test items i.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k)where k is an integer, a weighted loss function increases in value. 78.The method of claim 70 wherein the stopping rule is to stopadministering test items after administering the n'th test item i_(n)if, given the hypothetical administration of the sequence of test itemsi.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k) where k is an integer, aweighted uncertainty measure decreases by less than a predeterminedvalue, the weighted uncertainty measure being with respect to the SPSafter the administration of item i_(n) and after the hypotheticaladministration of the sequence of test items i.sub.(n+1), i.sub.(n+2), .. . , i.sub.(n+k).
 79. The method of claim 70 wherein the stopping ruleis to stop administering test items after administering the n'th testitem i_(n) if, given the hypothetical administration of the sequence oftest items i.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k) where k is aninteger, a weighted distance measure increases by less than apredetermined value, the weighted distance measure being with respect tothe SPS after the administration of item i_(n) and after thehypothetical administration of the sequence of test items i.sub.(n+1),i.sub.(n+2), . . . , i.sub.(n+k).
 80. The method of claim 70 wherein thestopping rule is to stop administering test items after administeringthe n'th test item i_(n) if, given the hypothetical administration ofthe sequence of one or more test items i.sub.(n+1), i.sub.(n+2), . . . ,i.sub.(n+k) where k is an integer, if a specified criterion issatisfied, the specified criterion being defined in terms of one or morefirst conditions, one or more second conditions, one or more thirdconditions, or combinations thereof, a first condition being that aweighted loss function increases in value, a second condition being thata weighted uncertainty measure decreases by less than a specifiedpredetermined value, and a third condition being that a weighteddistance measure increases by less than a specified predetermined value,the weighted uncertainty measure and the weighted distance measure beingwith respect to the SPS after the administration of item i_(n) and afterthe hypothetical administration of the sequence of test itemsi.sub.(n+1), i.sub.(n+2), . . . , i.sub.(n+k).
 81. The method of claim42 wherein one or more test items at the ends of a specified strategytree are removed if the weighted loss in administering test items forthe resulting strategy tree is less than the weighted loss for thespecified strategy tree, the weighted loss being obtained by weighting aloss function over all paths in the strategy tree and all test subjectstates, the loss function being a measure of the loss associated withadministering the test items in a path of the strategy tree.
 82. Themethod of claim 81 wherein the loss function is a function of (1) thestate of the domain, (2) a classification decision action that specifiesa state, and (3) the number of items administered.
 83. The method ofclaim 81 wherein the loss function consists of two additive components,the first component being a measure of the loss associated with theclassification of the test subject after administering the k test items,the loss associated with an incorrect classification being higher thanthe loss associated with a correct classification, the second componentbeing the cost of administering the k test items.
 84. The method ofclaim 83 wherein the first component of the loss function is (1) aconstant A₁ (s) if the test subject would be classified correctly afteradministering k additional test items and (2) a constant A₂ (s) if thetest subject would be classified incorrectly after administering kadditional test items, the constants A₁ (s) and A₂ (s) having a possibledependence on the state s, the second component of the loss functionbeing the sum of the individual costs of administering the k additionaltest items.
 85. The method of claim 42 wherein the decision rule is toselect the state associated with the highest value in the SPS.
 86. Themethod of claim 42 wherein the decision rule is to select the stateassociated with the smallest value for a weighted loss function.
 87. Themethod of claim 42 wherein the step of determining the classification ofthe test subject includes the step:administering test items to a testsubject in accordance with the strategy tree; classifying the testsubject in accordance with the path followed by the test subject throughthe strategy tree.
 88. The method of claim 1 wherein the domainspecifying step comprises the steps:specifying at least one initialmodel of the states in the domain; administering the test items in thetest item pool to a plurality of test subjects, classifying each testsubject as to state for each initial model; identifying superfluousstates in each model and eliminating them from the model; identifyingmissing states in each model and adding them to the model.
 89. Themethod of claim 1 wherein the test-item-pool-specifying step comprisesthe steps:specifying at least one initial model of the states in thedomain; administering the test items in the test item pool to aplurality of test subjects, determining for a test item in the test itempool the weighted frequency and/or probability that the test item isadministered; removing the test item if the weighted frequency and/orprobability is less than a predetermined value.
 90. The method of claim1 wherein the domain specifying step comprises the steps:specifying atleast one initial model of the states in the domain; administering asequence of test items from the test item pool to each of one or moretest subjects, the corresponding sequence of responses obtained for eachof the test subjects being called a test subject response pattern;determining the ideal response pattern for a test subject in each of oneor more domain states for each administered sequence of test items usingthe class conditional densities associated with each test item, an idealresponse being a value or a set of values; identifying those idealresponse patterns that do not satisfy a specified criterion with respectto each test subject response pattern, the specified criterion beingspecified in terms of one or more distance measures, a distance measurebeing a measure of the differences between a test subject responsepattern and an ideal response pattern.
 91. The method of claim 90wherein a state is removed from the initial model if its correspondingideal response pattern does not satisfy the specified criterion withrespect to a specified number of test subject patterns.
 92. The methodof claim 90 wherein one or more states is added to the initial model ifa specified number of ideal response patterns do not satisfy thespecified criterion with respect to one or more test subject responsepatterns.
 93. The method of claim 1 wherein the test-item-poolspecifying step comprises the steps:determining the intersections of thepartitions of states by the test items in the test item pool.
 94. Themethod of claim 93 wherein the test-item-pool specifying step includesthe step:adding new types of test items to the test item pool to reducemulti-state intersections to single-state intersections.
 95. The methodof claim 1 wherein the domain-specifying step comprises thesteps:determining the intersections of the partitions of states by thetest items in the test item pool; replacing the original domain modelwith a new domain model with new states, the new states being theintersections of the partitions of the original states by the testitems, the higher-lower-neither relationships of the new states beingderived from the higher-lower-neither relationships of the originalstates.
 96. The method of claim 1 wherein theclass-conditional-density-specifying step comprises the steps:(a)specifying prior distribution functions for the parameters of each testitem and state, the class conditional densities for the test items beingdeterminable from the item parameter distribution functions; (b)administering a sequence of test items from the test item pool to eachof a plurality of test subjects and recording the sequence of responses;(c) updating the SPS for each response in the sequence of responsesusing the initial SPS and the test item class conditional densities; (d)determining the test subject's classification; (e) updating thedistribution functions for the parameters of each test item and stateutilizing each test subject's classification; (f) repeating steps (c),(d), (e), and (f) until the process converges.
 97. The method of claim 1wherein the test-item-pool-specifying step comprises thesteps:determining the sharpness of a test item from the test item pool,sharpness being a measure of the capability of a test item todiscriminate between test subjects in different states, sharpness beingmeasured by use of one or more discrepancy measures; removing the testitem from the pool if its sharpness does not satisfy a predeterminedcriterion.
 98. The method of claim 1 wherein theclass-conditional-density-specifying step comprises thesteps:identifying test items having questionable class conditionaldensities, a questionable class conditional density being indicated by asharpness criterion not being satisfied; changing the class conditionalprobability density of one or more test items to achieve greatersharpness.
 99. The method of claim 98 wherein theclass-conditional-density-specifying step further comprises thesteps:removing from the test item pool any test items for which theclass conditional densities were changed and the sharpness criterion wasnot satisfied.
 100. The method of claim 1 further comprising thestep:specifying a remediation program for each state in the domain, aremediation program for state X being a compilation of facts associatedwith one or more other states in the domain and a procedure for teachingthe facts in the compilation to a test subject, the compilation notincluding facts associated with state X.
 101. The method of claim 1wherein the step of determining the classification of the test subjectincludes the steps:(a) administering one of a sequence of test itemsfrom the test item pool to the test subject; (b) updating the SPS afterreceiving a response to the administered test item; (c) applying astopping rule, steps (a) and (b) being repeated if the stopping rule sospecifies; (d) classifying the test subject as to a state in the domainin accordance with a classification rule if the stopping rule sospecifies; (e) directing the test subject to a remediation program.